OOCDHBHT BESOHE 



ED 145 816 IB 005 175 

TITLE A Survey and Analysis .of Hilitary Coaputer-Based 

Systems: A Tiro Part Study. Vpluae 11. A Descriptive 
and Predictive Hodel for Evaluating Instr iictional^^ 
Systems. Final Report. 

INSTITOTION McDonnell Douglas Astronautics Co. - East^ St. Louis, 

Mo. 

SPONS AGENCY Advanced Research Projects Agency (DOD) ^ Washiagton^ 

D.C. 

HDC-E1570 
15 May 77 
MDA--903-76-C-0086 

125p.; For related documents, £;ee l^D 135 401 (Volume 
if; Appendires A and' B have been removed due to 
marginal legibility. They are not ^included in the 
pagination 

MF-$0.83 HC-$6.01 Plus Postage. 

♦Computer Assisted Instruction; ♦Cost Effectiveness; 
Educational Innovation; Instructional Design; 
♦Instructional Systems; Linear Programing; ♦Military 
Training; ♦Models; Multiple Regression Analysis; 
Predictive Measurement; Training Techniques 
Method of Designing Instructional Alternatives 

This is the second volume of a tuo volume study. The 
first volume examined the literature to identify authoring aids for 
developing instructional materials, and to identify information 
clearing houses for existing materials. The purpose of »this volume 
was to develop a means for assessing the cost versus expected 
benefits of innovations in computer-based training systems. A 
three-pronged approach was used to address the problem. First, a 
descriptive model of a generalized, computer-based training system 
was developed. Second, a predictive model , was generated to describe 
student performance in a co«puter-based training environment. Third a 
computer program incorporating the predictive model was developed to 
estimate the cost of implementing one particular training innovation, 
i.e., revision of a conventional instructional program into a 
self-paced instructional program. A linear prediction model was 
developed to predict the time required for a student to complete a* 
unit of self-paced instruction, thereby demonstrating the time saved 
by converting from a conventional instructional mode. This was 
combined with the Rand Corporation MODIA cost analysis program. The 
study demonstrated that a viable time savings and cost model can be 
developed, ^iven a sufficient data base from which to draw the 
netessaty course content and student characteristics descriptions. 
(Author/DAG) 



REPORT NO 
POB DATE 
CONTRACT 
NOTE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 



Documents acquired by ERIC include many informal unpublished materials not available from other sourt^;,, makes every 

effort to obtain the best copy available. Nevertheless, items of marginal reproducibility are often encountered u.... da. ^ifects the 
quality of the microfiche and hardcopy reproductions ERIC makes available via the ERIC Document Reproducti^ja Gerviue (EDRS). 
is not responsible for the quality of the original document. Reproductions supphed by EDRS are the best that uau made from 



us DEPARTMENT OFHEALTH, 
EOUCATION»WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS DOCUMENT HAS OEEN REPRO. 
DUCED EXACTLY AS RECEIVED FROM 

STA^Fn no"'"'^ °'' 0 °'N IONS 

l^«To° °° "ECESSAHILY REPRE- 

SENT OFFICIAL NATIONAL INSTITUTE OF 

EDUCATION POSITION OR POL,Cv '° COPY NO. i£Z_ 



A SURVEY AND ANALYS8S OF MILITARY 
COMPUTER-BASED TRAINING SYSTEMS: 
(A TWO PART STUDY) 



15 MAY 1977 VOLUME II MDC E1570 

A DESCRIPTIVE AND PREDICTIVE MODEL 
FOR EVALUATING INSTRUCTIONAL 
SYSTEMS 

THIS RESEARCH WAS SUPPORTED BY THE ^ 
DEFEKSE ADVANCED RESEARCH PROJECTS AGENCY 
UNDER ARPA ORDER NO.: 3076 
CONTRACT NO.: MDA 903 76 C 0086 

The views and conclusions contained in this document are those of the author and should not be 
interpreted as necessarily^epresenting the official policies, either express or implied, of the 
Defense Advanced Research Projects Agency or the United States Government. 

SamtLouiS. Missouri 63166 (314) 232 0232 

/ 



MCOOrn/MELL DOUGLAS 




Unclassified 



SECURITY CLASSIFICATION OF THIS PAGE (When Data Entered) 



MDC El 570 



REPORT dOCUMEHTATION PAGc 


READ INSTRUCTIONS 
BEFORE COMPLETING FORM 


Ktr'WK I NUMdcR 2. GOVT AwwtbblvJN r*\J* 

MDC El 570 


<j • r\&w*r^l&r« 1 o wAi M ^ w r% \j rn t# 


4. TITLE (and Subtitle) 

A Survey and Analysis of Computer-Based Systems: 
A Two Part Study. Volume II. A Descriptive and 
Predictive Model for Evaluating Instructional 

Sy^tpm<; 


5. TYPE OF REPORT & PERIOD COVERED 

Final Report 

1 Oct 1977 to 15 May 1977 


6 PERFORMING ORG. REPORT NUMBER 


7. AUTHORC*; 

N/A 


0. CONTRACT OR GRANT NUMBERC*; 

MDA 903-76-C-0086 


9. PERFORMING ORGANIZATION NAME AND ADDRESS 

McDonnell Douglas Astronautics Company - East 
P.O. Box 516 

St. Louis, Missouri 63166 


10 PROGRAM ELEMENT. PROJECT, TASK 
AREA A WORK UNIT NUMBERS 


n. CONTROLLING OFFICE NAME And ADDRESS 

Defense Advanced Research Projects Agency 
Cybernetics Technology Office, 1400 Wilson Blvd. 
Arlington, Virginia 22209 


12. REPORT DATE 

15 May 1977 


13. NUMBER OF PAGES 

243 


U. MONITORING A'^EnCY NAME & ADDRESSf/f different from Controlling Offtcf) 


15 SECURITY CLASS, (of this report) 

UNCLASSIFIED 


15rt. DECLASSIFICATION DOWNGRADING 
SCH EDULE 


16. DISTRIBUTION ST ATEMENT Cof fh/s Roporf) 

Approved for public release; distribution unlimited 


\7. CJSTRIBUTION STATEMENT (of the abstract enterod in Block 20, H different from Report) , 

t 


18. SUPPLEMENTARY NOTES 


19. KEY WOJ^DS (Continue on reverse side if necessary and identify by block number) 

Computer-Based Training Model; Computer-Based Training System; Computer Models; 
Cost Model; Course Development; Data Base Development; Descriptive Models; 
Instructional Models; Instructional Systems; Instructional System Development; 
Linear Models; Multiple Linear Regression Analysis; Predictive Equations; 
Predictive Performance Models: Time-Savings Moduli Tr;^im'ng rnnn\/Atinn<; . 


20. ABSTRACT (Continue on reverse side if necessary and identify by block number) 

The objectives of this study were: (1) to examine the literature to identify 
authoring aids for developing instructional materials and to identify informa- 
tion clearing houses for existing instructional materials; and (2) to develop 
a generalized, computer-based model of a self-paced instructional system that 
can be used to determine expected time-savings and costs of self-pacing as com- 
pared to conventional instruction. The purpose of Part 1, reported in Volume 
I, was to examine the ISD concept as it relates to the authoring process and 
analyze tools and procedures which facilitate the ISD process, the purpose of 



ERIC DO .'?n"3 1473 BD,T,o.or 



I NOV 65 IS OBSOLETE 

i^j 



Unclassified 



'A 



SECURITY CLASSIFICATION OF TH15 PAGE (\i'hcn Data Entered) 



ERIC 



Unclassified 



SECURITY CLASSIFICATION OF THIS PAGE(tKhan Detm Enlerad) 



MDC El 570 



20. (Continued) 

Part 2, published in Volume II, was to develop a means for assessing the cost 
versus expected benefits of innovations in computed-based training systems. A 
three-pronged approach was used to address the problem. First, a descriptive 
model of a generalized, computer-based training system was developed. Second, a 
predictive model was generated to describe student perfarmance in a computer- 
based training environment. Third, a computer program incorporating the^pre- 
dictive model was developed to estimate the cost of implementing one particular 
training inribvation, i.e., revision of a conventional instructional program into 
a self-paced instructional program. A linear prediction model was developed to 
predict the time required for a student to complete a unit of ^elf-paced instruc 
tion thereby demonstrating the time saved by converting from a conventional 
instructional mode. This was combined with the Rand Corporation s MODIA cost 
analysis program. The study demonstrated that a viable time savings and cost 
model can be developed given a sufficient data base from which to draw the 
necessary course content and student characteristics descriptions. 



Unclassified 



SeC^^ljTY CLASSIFICATION OF THIS PAGEftW>en Data EnU!ed) 



EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUME II 



MDC El 570 
15MAY 1977 



PREFACE 



This study was conducted under Contract MDA-903-76-C-0086 with the Department 
of Defense, Advanced Research Projects Agency, Cybernetics Technology Office. The 
DoD"Program Manager was Dr. Harold F. O'Neil, Jr. The Principal Investigator was 
Dr. Robert A. Vecchiotti, McDonnell Douglas Astronautics Company - East (MDAC-EAST). 

The .contributors to this report are listed in alphabetical order: Dr. Donald 
W. Carver, Mr. Edmund H. Conklin, Drs. Donald R. Hesser, Wilson A. Judd, "Barbara 
L. McCofflbs, George D. Siering, and Robert A Vecchiotti. The contributors wish to 
express their appreciation for the cooperation and suggestions of personnel from 
the Air Force Human Resources Laboratory - Technical Training =it Lowry Air Force 
Base, Colorado, and to their colleagues on the Air Force Advanced Instructional 
System Program for their contributions, particularly Mr. William H. Stobie. Special 
thank? are due to Ms. Polly Carpenter-Huffman of the Rand Corporation for arranging 
the use of the Method of Designing Instructional Alternatives (MODIA) cost model. 



o 



i V 

I/-. 5 



ERIC 



. INSTRUCTIONAL SYSTEMS VOLUMEJi , 15 MAY 1977 

SUMMARY 

PROBLEM ■ . ■ 

Within the context of military training system development, the question of how 
to assess the cost versus expected benefits of innqvations in training technology 
has become of major importance. Before this question can be answered, it must first 
be demonstrated that both the cost and the tjenefits of such innovations can be quan- 
titatively measured. Then it must be demonstrated that accurate predictions can be 
made with regard to development and operational costs, tra-'ning effectiveness, 'and 
expected .timesavings. 

APPROACH 

Three separate activities were undertaken to solve this problem. The first 
was to define a descriptive model of a generalized, computer-based training system. 
This model defined the domain of such systems to be addressed by a predictive model. 
It also identified the component activities involved in such a training system, 
isolated seven clusters of factors v/hich would impact the operation of the system, 
and identified the relationship of each cluster to the separate component activities. 

The second step was generation of a predictive model of student performance in 
a computer-based training environment. The purpose of this model was to predict 
the time required for a student to corrplete an arbitrary unit of instruction in a 
self-paced training program given only characteristics of the student, of the content 
taught, and of the method employed tc teach the content in a conventional, lock-step 
course. 

The third activity was to develop a computer program which incorporated the 
predictive model and estimated the cost of implementing and operating this particular 
innovative training technique, i.e., conversion from conventional lock-step to self- 
• paced, computer-based instruction. 

RESULTS 

A linear prediction model was developed to predict a student's time required to 
co.nplete his first attempt on a given lesson. Fourteen predictor variables were 
selected, eleven descriptive of the instructional course content and three descrip- 
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tive of student characteristics. The final regression equation produced on R = .63 

accounting for 39 percent of the variance in a student's first-attempt-lesson time. 

2 

When the regression equation was validated on a second sample, the R resulted in a 
loss of only 5.34 percentage points in accounting for the variance in the criterion 
variable, 1 irst-attempt-lesson time. The basic model was then modified to predict^ 
total course completion time. 

A computer program, "Aid for Instructional Development and Evaluation (AIDE)," 
was written. It is a modification of Rand Corporation's MODCOM cost analysis program. 
It retains all of the cost-analysis features and outputs of MODCOM while allowing 
course completion time to be estimated by the regression equation developed in this 
study. Course completion times and cost estimates were generated for a standard 
sample and three additional samples in which student or course content characterise, 
tics were varied. 



^CONCLUSIONS 

This study demonstrated that a viable model can be developed which predicts 
the costs and timesavings associated with individualization of an arbitrary training 
course through the use of a computer-based training system. The generality and pre- 
cision of such a model is dependent on the breadth and accuracy of the data base from 
which the model is derived. Although the data base employed in this study was not 
as complete as could be desired, the resultant model was found to be reasonably 
accurate and is sufficiently general to be of value in the evaluation of courses for 
individualization. Mo^^e importantly, a methodology has been developed and demon- ' 
strated which will become an even more useful tool as additional data are accumulated 
from computer-based training systems. 

The results of exercising the AIDE program illustrated the impact of variations 
in course content on the timesavings to be expected from individualization. Current 
methods of instruction, methods employed in the course as it is taught in conventional, 
lock-step mode were found to be sensitive indicators of variations in course content 
and, hence, of expected timesavings. Thus, performance oriented training, for which 
the Plan of Instruction indicates a relatively high proportion of demonstration- 
performance activities, presents less^ promise for individualization than do courses 
containing a high proportion of cognitive and, particularly, memorization content. 
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1.0 INTRODUCTION 

1.1 BACKGROUND 

A -question of primary importance to those concerned with the design, 
. development, implementation, or evaluation of innovations in training technology 
is one which addresses the cost and effectiveness of such innovations. Within the 
context of military training system development,, interest has recently been focused 
on this question as it relates to innovations in the area of computer-based 
training {C.^). Here the concern has been one of not only attempting to find means 
for evaluating the actual benefits attributable to the innovation relative to its 
cost, but also to find a means for evaluating the expected benefits. That is, 
before large sums of mone> are invested in computer-based training innovations, it 
is important to know (a) whether the benefits and costs associated with the 
innovations can be quantitatively evaluated and (b) whether cost-effectiveness 
data to support implementation decisions can be predicted, given quantifiable 
information on training system parameters and their interactions with trainee and 
training material characteristics. 

^ To address the preceding concern adequately^ both descriptive and predictive 
models of i computer-based training system are required. Descriptive models 
serve the purpose of identifying the relevant parameters affecting cost and effecti- 
veness decisions and of indicating the degree^to whicii these parameters may be 
quantified. Predictive models serve the purpose of calculating expected outcomes, 
given information on the descriptive parameters and the relationships between them. 
A major problem, however, is that appropriate descriptive and predictive models 
have not been sufficiently integrated in an operational decision model which can 
be used in planP"** g cost-effective implementations of CBT innovations. 

The present study was undertaken to address this need. The specific goals and 
objectives of the effort are described in the following section, after which the 
types of problems involved in the development, integration, and validation of 
descriptive and predictive CBT models are discussed in a literature review. 
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1.2 GOALS AND OBJECTIVES ' \ ^ 

The goals of the present study were threefold: (1) to define and describe 
those computer-based training system parameters likely to affect cost-effective- 
ness decisions, i.e., to produce a descriptive model of a generalizable CBT system, 
including all relevant instructional, student, and training system variables^; 
(2) to model, as accurately as possible, anticipated student performance In the 
generalized system described; and (3) to produce an operational computer model 
which will incorporate the predicted student and system performance data and can 
be used to calculate the expected costs and benefits of implementing selected CBT 
innovations. 

To meet the preceding goals, the to! lowing objectives were specified: 
a. To describe the CBT characteristics expected to be related to measure- 
able outcomes. 

• j^^- Jo-Identify those outcomes that most reflect "benefits" attributable to 
innovations, e.g., student training time reduction and decreased elimination 
rates. 

c. To specify the relationships and interactions between all system 
parameters. 

d. To identify those parameters for which quantifiable data are available. 

e. To determine the type of modeling procedures most appropriate for the 
predictive model, e.g., appropriate statistical or mathematical models for 
predicting benefits and costs. 

f. To integrate the predictive model in a computer program that can b^} 
used as a planning aid. 

The anticipated and actual problems related to these goals and objectives 
are discussed and summarized in the following section. 

1.3 LITERATURE REVIEW 

1.3.1 Selection of Criterion Variable - Only a few pertinent reports con- 
cerning the quantification of learning system features m tenns of time-to- 
complete a unit of instuction, student performance, and training cost were identi- 
fied. A seminal study reported by Wagner, Behringer, and Pattie (1973) addressed 
the description of training system features which predicted course completion time. 
Their study combined course-related Instruments and standardized aptitude tests 
to predict time-to-complete. These devices included: 
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0 Tests of skills and knowledge necessary for the acquisition of certain 

military occupational specialties 
0 /Tests that <^ampled the content of the tasks to be learned d»'^ing the 
' course 

0 Scales designed to measure motivational factors related to the subject 
- matter of th'^ "^se. 

A step-v/ise multiple linear regression analysis was applied to the data. Correla- 
tions between the predictor variables and course completion time ranged from .65 
to. .87. These results supported the hypothesis that course-rela,ted predictors 
are better than measures of general aptitude in estimating the time- to- complete 
a course of instruction. In addition, the authors accepted the hypothesis that 
the relationship between course-related predictors and completion time variables 
is linear. 

^ As an attempt to categorize and Integrate the diverse variables influencing 
the instruQti^onaKprocess, Carroll^ (1962, 1963) presented a conceptual model of 
school .learning which has proven to be quite durable over the intervening fourteen 
years. The model output, the, criterion vat^iable, is degree of learning, i.e*, 
performance on an achievement test after a fixed amount of e,lapsed time. The 
model itself consists of five elements, two pertaining to the characteristics ^ 
of the instructional environment, and three to the characteristics of the student. 
The first instructional variable concerns the adequacy of ^presentatio n of the 
learning task, how clearly it is presented and explained, and how appropriately it 
is placed in the sequence of tasks to be learned. The second represents the 
opportunity for learning , the time allowed to learn the task. The first of the 
three individual difference variables represents the student's ability to" understand 
the Instruction . Carroll suggests this to be analogous to ger^eral or verbal 
intelligence and determines the extent to which the student is able to understand 
directions and explanations or to infer them from the content of instruction if- 
they are missing. The second, aptitude , is defined as the time needed by the student 
to learn a particular task to criterion, giv(ri that the instruction is presented 
well enough for him to grasp it. Aptitude, is considered to be a relatively invarient 
characteristic of the individual. The third element represents the maximum amount 
of time an individual will apply himself to a task This is called perseverance . - 
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A recent model of the instructional process proposed by Cooley and Lohnes , 
(Glaser, 1976) deviates only slightly from CarrolTs earlier formulation. Carroll's 
elements of aptitude and ability to understand instruction are replaced by a single 
variable, initial ability , representing the 5um of the student's skills and intel- 
lectual capacities, Carroll's adequacy of presentation is replaced by two elements: 
(a) the extent to which thfe curriculum is structured, the methods of individuali- 
zing instruction, and the general organizatioa of instruction; and (b) the quality 
of the s. dent-teacher interaction. The term motivation has been substituted for 
perseverance , but it is defined as the studentjs tendency to engage in learning 
activities when the opportunity exists". Finally, the relationships among the 
elements are not stated as explicitly. Rather, these relationships are to be 
determined empirically through multivariate analysis. 

Neither the Carroll nor the^'Cooley and Lohnes model, as stated', is appropri- 
ate to d self-paced, criterion-referenced instructional environment. In both, the 
criterion variable represents degree of learning, and opportunity for learning 
is assumed to be limited. In a criterion-referenced environment, degree of 
learning is, theoretically at least, controlled while opportunity for learning 
remains free to vary. Consequently, t-^-^e per unit of instruction (time to 
criterion) or learning rate is the most appropriate criterion variable for a model 
of learning in a self-paced environment, 

♦ 

1,3,2 Selection of Predictor Variables - Psychologists and educators have 
long been aware that learning and performance on instructional tasks, are a func- 
tion of the student's level of ability or aptitude and his motivation. More 
re/ently, they have also recognized the importance n^f taking into account both 
iituational and task factors. The search )for those variables, however, and 
'their associated measures which best describ,e the student's actual and expected 
performance remains a pervasive problem fi/rther complicated by: 

0 The difficulty in obtaining quantitative measures of situational and 
task variables (Uprichard, 1975) 
^ 0 The fact that a student's trait levels of ability and motivation may not 

be as predictive of his performance as more si tuational ly sensitive state 
measures (Spei'lberger , 1966; Spielberger, O'Neil, and Hansen, 1972) 
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0 The limited progress that has been made In the identification of ".atis- 
factory measures of student motivation (Bem and Allen, 1974). 
Additional problems are raised when the concern is prediction of performance 
in a self-paced, individualized, criterion-referenced instructional environment. 
A majority of research on variables related to learning has been conducted in 
group-paced, conventional instructional environments, (Campbell and Schwen, 1971; 
Schweii, 1973), and existing theories and data on ability and motivational 
variables have focused on achievement scpres rather than learning rates as the 
dependent variable of interest (Brown and James, 1972; .Colton, 1974; McAvoy, 
Kalin, and Franklin, 1973). , 

1-3.2.1 Student Ability /Aptitude - A conmion approach to the problem of 
identifying student characteristics related to performance in a complex instruc- 
tional system is some form of task analysis. Campbell and Schwen (1971) argue ^ 
for an analysis of learning tasks in terms of the effects they have on the student's 
cognitive processes and personality variables. They stress that possible inter- 
actions between task and learner characteristics be considered. Schwen (1973) 
suggests that a model most appropriate for analysis of learner characteristics 
is one that combines empirical methods, e.g., predictive statistical models, 
with task analysis. The important point is that such analytical methods can be 
used to identify potentially important variables or interactive relattonships- 
between variables that should be considered in a student ability/aptitude para>l 
meter. 

1'3.2.2 S ituational/Task Variables - In considering relevant situational/ 
task variables, it is instructive to distinguish between those variables which may 
vary between students, such as type of instructional treatments, and those which 
are constant for an individual student but var> across tasks, e.g. , criterion 
levels or task difficulty. Just as analyses of learning tasks and learner " 
characteristics were recommended for the identification of an ability/aptitude 
parameter, it is an equally appropriate approach to the identification of 
relevant situational/task variables (Campbell and Schwen, 1971; Schwen, 1973). 
Walberg (1975) has stressed the need for analyses which identify other possible 
variables predictive of student performance. He bases tj)is on the fact that, 
aHhough aptitudes or abilities can be reliably measured, they can usually account 
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for no more than half the variance in student performance. Uprichard (1975) 
suggests that organizational stucture or classroom variables be taken into account 
as well as evaluation variables, e.g.> criterion levels and frequency of tests. 

Some additional situational/task variables which may be important in a self- 
paced instructi^JfiaT environment are suggested by the Wagner et a1 (1974) 
study. Among their findings were: 

0 Total time could not be predicted as well as instructional time due to 
many unknown factors related to variabilities in testing time 

0 Grouping of data by instructional mode was required in order to predict 
completion times more accurately. 
These findings suggest that potentially important si tuatioVial/task variables would 
include ratios of instructional time to testing time, relative difficulty level 
of each i.nstructional unit of interest, testing frequency, ratio of audio-visual^ 
to printed materials, and the frequency with which an adaptive model r'-jice 
(if present in the system) constitutes the actual assignment. 
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2.0 OVERVIEW OF THE TECHNICAL APPROACH * 

The work described herein consists of three major activities: 

a) Definition of a descriptive model of a generalized, computer-based 
training -system. 

b) Generation of a predictive model of student performance in a computer- 
based training environment* 

c) Development of a computer program incorporating the predictive per- 
formance model to estimate the cost of such a system. 

The, primary purpose of the descriptive model was to identify the component 
activities involved in the instructional process in ^ computer-based training 
environment. Further, the model served to identify: (a) subject matter charac- 
teristics; (b) instructional and system design considerations; (c) student indi- 
, vidua! difference characteristics; and (d) the relationship of these variables to 
the various components of the model. The variables to ^be considered by the pre- 
dictive model served to define the domain of computer-based training systems to be 
^ addressed by the predictive model* As defined, the model assumed a criterion 
referenced testing environment and emphasized, as the dependent variable, the time 
to reach criterion on an arbitrary unit of instruction* 
•» 

Since this study focused on the prediction of student time-to-completion of a 
unit of instruction (so as to demonstrate time-savings and cost-effectiveness), ^ 
a model to account for predictiori of time was developed. The analysis of data 
within this model necessitated the use of a statistical technique through which 
one can analyze the relationship between the criterion or dependent Variable 
' and a set of predictor or independent varial:)les. 

The statistical technique chosen for this effort was multiple regression. 
This statistical treatment is useful primarily because of its ability to produce 
linear prediction equations (Kerlinger and Pedhazer, 1973) as well as computing 
statistics that provide the researchers with statistical inference procedures 
for determining confidence limits for estimates with a means for measurement of 
variance in the dependent variable accounted for by the predictor set* Finally, 
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the use of multiple regression provided the authors with a degree of control over 
extraneous variables in order that the combined contribution of a specific 
set of variables could be inore closely examined. 

Specifically, the intent was to' develop a model in the form of a regression 
equation in which the criterion was time-to-complete an arbitrary instructions! 
unit. Predicto'' variables were drawn from the several categories defined "by the 
descriptive model and were limited to the type of information which will be 
available to or can easily be obtained by a military training course developer. 
The actual predictor variable data were sampled from course planning documents 
for two courses which had been group-paced but are currently being conducted in 
a self-paced mode under the control of a computer-based training system ~ - 
the Advanced Instructional System (AIS) at Lowry Air Force Base, Colorado. 

• The predictor variables fell into four broad categories. The first concerned 
course content, information which was either available directly from the course 
planning documents or which could be derive.d from these documents. The second 
category consisted of instructional design considerations. Variables in this 
category were broadly defined and limited by the consideration of only those 
which' varied between or within the two AIS courses. The third consisted of test 
characteristics, the performance standards established for the two courses. The 
fourth category of predictor variables consisted of student individual ^difference 
measures. Two classes of such measures were considered: those which would 
normally be available to the developers of a military training course; and an 
additional set of measures which was available to this study because of its use 
in the AIS pre-assessment battery. 

The criterion variable explicitly predicted by the regression equation is 
first-attempt-time on an AIS lesson. The model was then expanded to address the 
time required to complete a larger instructional unit consisting of ii lessons and an 
end-of-unit criterion test./ Lesson and unit level criterion data were drawn from 
the AIS student performance data bases for the Inventory Management and Weapons 
Mechanic courses. 

The most relevant predictors in each of the four categories of predictor 
variables were determined through a series of linear model comparisons. In such 
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comparisons, two equations, one including and the other excluding the variable(s) 
of interest within variable categories, are evaluated in terms of the reduction in 
the square of the multiple R from the first to the second linear model. In the 
comparisons made, primary emphasis was placed on the cost of obtaining data for 
the variable(s) being considered for exclusion relative to its (their) predictive 
value. Following reduction to a parsimonious set of predictors, stepwise multiple 
.linear regression was used to obtain a final predictive model. This final model 
was then cross-validated an J tested for generality as an additional evaluation 
of validity across courses. 

The final activity in the study was the production of a computer cost analysis 
program entitled Aid for Instructional Development and Evaluation, (AIDE). AIDE is 
a modification of the Rand cost analysis program MODCOM, Method of Designing 
Instructional Alternatives (MODIA) (Hess and Kantor, 1976) set of programs. The 
modification consisted of re prog ramming MODCOM to use the predictive model regres- 
sion equation developed in this study while retaining all of MODCOM's cost analysis 
features. The program output is a , series of tables which describe the course 
specifications including the student and lesson variable values employed by the 
predictive model regression equation, resource requirements and costs, and the 
total course cost. To illustrate its use, the AIDE program 'was exercised in 
answering three sample questions related to student characteristics and course, 
design (See Section 5.4). 
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3.0 ^DESCRIPTIVE MODEL OF COMPUTER-BASED TRAINING SYSTEMS 

Antecedent to the development of the computer-based predictive model, it was 
necessary to define the scope of the instructional process to be modeled. This 
was done through^ the generation of a descriptive model. This descriptive model 
provided a' basis for (a) conceptualizing the overall instructional process, (b) 
identifying gaps in the conceptualization, (c) identi-fying variables which could be 
quantified for inclusion in. the predictive model, (d) determining data collection 
requirements, and (e) assistj'ng in the interpretation of results. Wtiat is pre- 
sented is not intended to represent the only way or the optimal method of 
accomplishing ind'^/idual ized, self-paced instruction, but rather it is an attempt 
to represent a generalized model which can encompass a wide variety of computer- 
based training systems. The descriptive model was developed in two stages. The 
first stage, essentially the context for the descriptive mo^Jel itself, is illustra- 
ted in Figure 1. 

The most basic element defining the characteristics of an ^instructional 
process model would appear to be the criterion behaviors to which the student 
is, to be trained - the set of skills assumed to be required in the field. In 
the Air Force, these skills are rather explicitly defined by the Specialty Training 
Standard (STS) for each field. 

The curriculum, che instructional content of a particular course, represents 
the interpretation of these behavioral 'goals by the course personnel. Since the 
curriculum is defined by the course personnel, it must be assumed that it will be 
shaped by their cumulative experience, both in the field and in the classroom, 
and by their knowledge of local institutional constraints, e.g., availability 
of training devices, usual ratio of instructional personnel to students, and physi- 
cal plant limitations. The final definition of the curriculum is represented by 
the course planning documents. For the Air Force, this is the Plan of Instruction 
(POI). These documents contain an explicit statement of the course content, the 
amount of instruction (time) to be devoted to each topic, tiie instructional method 
(e.g., lecture or demonstration) where available, and the training aids (e.g., 
films, manuals and actual equipment to be employed). Throughout thu^t-vivij it. 
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FIGURE 1: General Model of Instructional System Development, Implementation and Evaluation 
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will be assumed that the course to be modeled is in existence, probably in a 
conventional, group-paced format, and that course planning documents are avail- 
able. 

Since the purpose of the predictive model being developed is to represent a 
self-paced, individualized course, it is assumed that in situations in which the 
model is to be employed, a complete set of instructional materials, representing 
the entire course content, will need to be selected or developed. Thus, the next 
blofck in the diagram represents the means by which the instructional content is to 
be presented. These are the defining characteristics of the course of instruction 
to be developed. Is the instructional method to be expository or oriented toward 
discovery and problem solving? Is the presentation narrative or programmed, that 
is, frame oriented with embedded questions? If progranmed, what is the frequency 
of interaction? To what extent are pre- and post-organizers, such as objectives 
and post-questions employed? Is the instruction merely self-paced, or is it indivi- 
dualized in the sense of advanced placement and assignment of alternative 
instructional treatments? What presentation media are employed, e.g., print, audio- 
, visual, or computer-assisted instuction? As is indicated by Figure 1, these 
instructional characteristics are a function of several considerations: the 
content to be taught; explicit decisions regarding instructional design, indivi- 
dualization strategies, and media; the availability of instructional resources; 
and the influence of institutional constraints and policies. The category of 
instructional resources is mean^t to include presentation media (printed materials, 
audio-visual devices, terminals for computer-assisted instruction), the materials 
and equipment which the student is being trained to work with in the field (e.g., 
manuals, actual devices and tools, simulators), and terminals for interacting with 
a computer-management system. 

The instructional interaction, the process through which learning takes place, 
is primarily a function of the instructional content, mode of presentation, and the 
characteristics of the students at the time of their entry into the xourse - the 
level of their aptitudes and abilities. Although not shown explicitly in Figure 1, 
student characteristics will, hopefully, have been taken into consideration by 
the instructional design process and, to the extent necessary, reflected in the 
course planning documents. It is unreasonable to expect, however, that the instruc- 
tional presentation will be perfectly matched to student characteristics. There 
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is also the possibility that the average ability level of the students will vary 
over time as a function of demand for trained personnel and economic fluctuations 
in the civilian sector. Finally, the total distribution of student charac- 
teristics, as well as the average level, must be considered in evaluating the 



It must be recognized that the nature of the instructional interaction is 
also shaped by existing course policies. These are, unfortunately, intangible 
factors concerning the prevailing attitudes among the instructors and their super- 
visors, and* the incentives, positive and negative, explicit and implicit, to which 
the students are exposed,' Finally, the instructional interaction, particularly the 
time it requires, is directly a function of the actual, day-to-day availability 
of the instructional and management resources. While resource availability was 
presumably taken into consideration by the instructional system design^, problems 
may ari.se if the student entry rate is substantially increased or if equipment 
maintenance is not adequate. In that case, student queuing for resources can 
account for a substantial' portion of the total training time. 

There is undoubtedly a host of measures by which the effectiveness of the 
instructional interaction, of the total instructional system, can be judged. 
The most tangible measures, however, are level of mastery, rate of attrition, 
and time-to-criterion. As will be discussed in subsequent sections of this report, 
level of mastery is exceedingly difficult to capture in a model intended to genera- 
lize across training programs. Since the prime purpose of the predictive model is 
to provide a tool for the comparison of alternative instructional systems, all 
of which are criterion referenced, it will be assumed that the required level of 
mastery is primarily, a policy decision that remains constant from one system to 
another. If level of mastery is indeed fixed, the only two parameters left free 
to vary are rate of attrition and tim^-to-cri terion. While attrition would appear 
to be a plausible dependent variable, it has been the authors' experience that, in 
a miltary training environment at least, it is also largely a function of course 
policy. Thus, the primary dependent variable to be considered by the model will be 
time-to-criterion, the tlrrie required to complete the course of instruction, 

'Given the dependent variable of time-to-completion, the next step was to 
develop a description of the instructional interaction process which would 
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(a) account for time-to-completion, (b) generalize across a variety of computer- 
based training systems, and (c) incorporate the classes of independent variables 
outlined in Figure 1 and the preceeding paragraphs. Finally, it was desirable 
that the model be sufficiently fine grained that each of the individual components 
of the model could be related to a small subset of the independent variables and 
that the times attributable to each component sum to equal total time-to-complete 
the instructional unit. This descriptive model is shown in Figure 2. 

A course structure is assumed in which the curriculum consists of one or more 
major instructional units. Each unit is assumed to end with a criterion referen- 
ced examination which the student must pass in order to proceed to the next unit, 
or, in the case of the last unit, to be graduated from the course. Irt the descriptive 
model, this examination has been termed a credentiall ing test. In Air Force tech- 
nical training, the corresponding instructional unit is a "Block." Student scores 
on end-of-block tests are entered in the student's permanent record; a student 
who meets criterion on the block test is considered to have demonstrated proficiency 
on the block content, and decisions concerning academic elimination from the , 
course are made on the basis of block te-st scores and times-to-block-completion. 
It is assumed that, up to a point, a student failing to pass the credentiall ing 
test will be reassigned part or all of the instructional unit and then retested. 
The descriptive model is intended to account for the time required for a student 
to complete an arbitrary instructional unit - to reach criterion on the unit 
credentiall ing test. 

Within each major instructional unit, it is assumed that the curriculum is 
divided into a number of smaller units, each covering one or more objectives, 
which will be termed lessons. It is further assumed that, in general, each lesson 
has a criterion referenced test over the lesson content. Lesson tests are intended 
to be for diagnostic rather than credentialling purposes. That is, the primary 
purpose of the diagnostic tests is to track student's progress through the course, 
to detect specific areas of weakness and misunderstanding, and to provide a basis 
for lesson level remediation in these areas. Diagnostic test scores would not, as 
a rule, be made part of the student's permanent record. 

Turning now to the components of the model, the first decision point. 
Determine Set of Assignments , represents a form of ijidivldual ization which. In 
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FIGURE 2: DESCRIPTIVE MODEL OF TIME TO COMPLETE AN INSTRUCTIONAL 
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any particular system, may be absent or present and, if present, may be present 
in varying degrees. It serves two functions. On a student's first pass through 
the instructional unit, it is analogous to. advanced placement ~ on the b^sis of 
known information about the student's abilities and current skills, is it pro- 
bable that he could skip one or more lessons without seriously jeopardizing his 
chances of meeting criterion on the credential 1 ing test? Similarly, on a remedial 
pass through the unit (following failure to meet criterion on the credential 1 ing 
test), what minimal set, of lessons (or partial lessons) can be assigned and still 
have reasonable assurance th'\t the student will meet the credentialling test cri- 
terion^on a retest? Within. the descriptive model, this component is represented 
by the probability (p^-) that the student will be assigned a given lesson (lesson i). 
It is assumed to be a function of the system's capability foe individualization, 
the content of lesson i, the characteristics of the credentialling test, and the 
student's specific abilities and skills. 

The second decision point. Make Next Assignnent ^ is similar in that it also 
represents a form of individualization which may be absent or preservt in varying 
degrees in different' instructional systems. It is intended to represent system 
capability for individualized assignment of alternative instructional treatments 
at the lesson level. Time savings attributable to this component are represented 
as being distributed across other components in the model. That is, individua- 
lized assi*gnment of alternative treatments will impact both first attempt and 
remedial study times, the probability of passing the diagnostic tests, and, to an 
extent, the time required for testing itself. 

The next component. Queuing for Lesson i Resources , is the first to represent 
a time consuming activity, any delay due to the unavailability of lesson resources 
which the student may encounter prior to being able to start his next assignment. 
It has lower limit o*f zero time and is assumed to be a function of both the quantity 
of available resources of the needed type and the system's capability for resource 
management, for scheduling assignments so as to avoid resource bottlenecks. 

The component entitled First Attempt ^tudy on Lesson i represents the amount 
of time, consumed by the student's initial study of the lesson i content on a given 

— , 
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pass through the instructional unit, either his first pass through the unit or 
unit level remediation* This component accounts for the major portion of the 
student* s first pass time in the instructional unit and is assumed to be a function 
of the lesson content, system capability for individualization (the availability 
and individualized assignment of alternative instructional treatments), instruc- 
tional design and presentation considerations, the known (to the student) 
characteristics of the lesson 1 diagnostic test, and the characteristics of the 
student himself. 

3 J FIRST ATTEMPT DIAGNOSTIC TEST TIME . 

The amount of time which the student spends completing the test following 
lesson i, has a lower limit of zero (representing the case in which there is no 
lesson i diagnostic test) and is ass'umed to be a function of the test characteris- 
tics. These characteristics include whether the content to be tested is primarily 
knowledge or involves. the demonstration of performance skills, and if written, 
the type of items (completion, ^multiple choice, or true-false), the number of 
items (or number of steps in a performance task), and the criterion level* While 
it could be argued that the amount of time the student spends on the test 
is also a function of his level of knowledge, and hence a function of the same 
factors that influence study time, the simplifying assumption is made, for the 
sake of parsimony, that these factors make a relatively small contribution to the 
total variance in testing time. They are taken into consideration in a subsequent 
componept representing the probability that the student will meet criterion on 
the diagnostic test. 

3.2 QUEUING TIME FOR LESSON TEST SCORING/TRACKING . 

This represents the amount of time the student must spend, following 
completion of his diagnostic test, until his test has been scored and he has 
received his next assignment. Placing this component after the diagnostic test 
time component implies that testing itself is done off-line. The position of the 
two components would simply.be reversed to represent an on-line testing situation. 
It is assumed % be a function of the number of terminals available for student/ 
system interaction, the speed with which these interactions can take place, and 
the extent to which assignment scheduling (resource management) can prevent the 
-occurrence of waves in the level of student demand for terminals. 

r ' ' 29 



EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUME II 

0 



MDC El 570 
15 MAY 1977 



3.3 PASS/REMEDIATE DECISION 

The following decision point represents the outcome of scoring the diagnostic 
test* the probability that the student has met the test's stated criterion, 
ir so, it is assumed that he proceeds to his next lesson assignment, if any. If 
not, it is assumed that he is assigned a remedial activity designed to enable him 
to meet criterion on the previously failed objectives. This probability is assumed 
to be a function of the content of lesson i, the system's capability for indivi- 
dualized assignment of alternative instructional treatments, instructional design 
considerations for lesson i, the characteristics of the lesson i. diagnostic test, 
and the relevant characteristics of the student himself. If there is no diag- 
nostic test corresponding to lesson i, the decision would, of course, always be 
"Pass." 

Let us first consider the case in which the student fails to meet criterion 
on the diagnostic test and is assigned a remedial activity. The bulk of this 
activity is represented by the component labeled Remedial Study on Lesson i . 
The amount of time required for this activity is assumed to be a function of the 
same factors which determined first attempt study time on the lesson and, in 
addition, course training policy considerations. The intent of including this 
policy factor is to allow for situations in which the local training policy 
dictates that the student retest within . a certain jDeriod of time or that the 
student spend at least a minimum period of time before retesting. 

It will be noted that the component Queuing for Lesson i Resources has not 
been included in the remedial loop. It is assumed that resource management will 
be such that if a training device was required for the student's first attempt on 
the lesson, the device will remain available to him for remediation. 

3.4 REPEATED DIAGNOSTIC TEST ON LESSON i AND QUEUING FOR LESSON TEST SCORING/ 
TRACKING 

The next two components, are essentially the same, and a function of the same 
factors, as the comparable components in the loop respresenting initial study. 
It should be noted that the number of items on the diagnostic test, and hence the 
time required for testin^g, could change if a scheme is employed by which a student 
is retestejd only over previously failed objectives. 
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The following decisior. point, the probability that the student will now 
meet criterion, is analogous to the prior, first pass decision point and is 
assumed to be a function of the same factors with the addition of a course policy 
variable. The introduction of the policy factor is to allow for those situations 
in which local policy might require that a student proceed to the next lesson 
assignment after two or three lesson attempts. This might be accomplished, for ^ 
example, through intensive remediation by an instructor and his certification chat 
the student has met the diagnostic test criterion. If the decision is that the 
student again failed to meet criterion, it is assumed that he will recycle through 
the remedial loop. A decision that the student has now met criterion returns him"^-^ 
to the same point as was the case for a student who met criterion following .his 
initial attempt on lesson i; • 



^.5 INSTRUCTIONAL UNIT LEVEL COMPONENTS . 

If the set of lessons assigned to this student for the current instructional 
unit has not yet been completed ( All Assignments Completed? ) , he repeats the cycle 
described In the preceding paragraphs. If all assigned lessons have been completed, 
that is, if the student has stuoied all assigned content in the instructional unit, 
he is assigned the unit credentiall ing test. 

The component shown preceding the credential ling test, Cumulative Administrative 
Overhead , is intended to account for that time which, while the student was in the 
current instructional unit, was spent on non-instructional activities. Examples 
would be roll call, extra group instruction such as safety lectures, classroom 
dean up, assignment to special details, smoking and coffee breaks. This time 
is to be strictly a function of local course policy and would have to be estimated 
in each specific situation. 

Time spent completing the credentialling testis assumed to be a function of 
the same test characteristics as was the, case for the diagnostic tests. This . 
component would, however, also include any time spent in critiquing the student's 
item-by-item performance on the credentialling test, an activity assumed to be 
unlikely following the more frequent diagnostic tests. 
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Again, allowance must be made for the time required to complete test scoring. 
This is represented by the component labeled Queuing for Unit Test Scoring/Tracking 
and, as was the case for diagnostic test queuing, is assumed to be a function of 
terminal resource availability. * ' • 

3.6 PASS/ REMEDIATE/WASHOUT DECISION 

The final decision point represents the outcome of the evaluation of the 
student's performance on the credential ling test. If the student is determined 
to have met the stated criteria, he proceeds to the next instructional unit or is 
considered to have completed the course. It is assumed that a student who fails 
to meet criterion will be "washed back" to repeat some or all of the lessons in 
,^ the instructional unit. For the sake of parsimony, it is assumed that this, major, 
unit-wide remediation can be modeled in the same manner as was the student's 
first pass through the unit. The amount of material assigned for remediation 
may differ from the student's first pass through the unit depending on the system's 
capability for individualization. Finally, it is assumed that the decision to 
eliminate a student from the course on the basis of academic deficiencies will , 
only be made following a credential 1 ing test. 
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4.0 PREDICTIVE MODEL OF COMPUTER-BASED TRAINING SYSTEMS 

.4.l/ DESCRIPTION OF DATA SOURCE 
/ 

4.1.1 Selection Cp'teria - The criteria established for selecting a data source 
were as follows: 

^ 0 A computer-based, self-paced, individualized instructional system well 
into its development cycle 
0 The availability of quantitative and qualitative course content descriptions 
from both the prior, conventional mpde of instruction and from the current 
self-paced mode 

0 The availability of pre-instructional measures of trainee characteristics 
0 The availability of trainee performance (time and score) data under the 
computer-based system. - 

Of the available sources, the Air Force Advanced Instructional System (AIS) 
at Lowry Air Force Base, Colorado, was considered the best to meet these' criteria. 
In addition, resident specialists, familiar with the characteristics and use of 
the AIS data base were available as consultants. Computer time and a limited 
amount of software support were also made available. 

4.1.2 The Advanced Instructional System - AIS is being developed by the McDonnell - 
Douglas Corporation under a contract with the Air Force Human Resources Laboratory 
(TT), Lowry Air Force Base (Rockway and Yasutake, 1974; Yasutake and Stobie, 1975). 
It is intended to be a comprehensive CMI/CAI system for the administration and 
management of large scale, individualized technical training. At thje time of this 
study, four courses, containing approximately 600 students, were being supported 
with the overall goal of achieving an average 25 percent reduction in training time 
over what was required prior to system implementation. These four courses represent 
a wide range of learning requirements, from clerical to mechanical to problem . 
solving skills, and cover a variety of course administrative requirements. In 
addition, AIS serves as a research and development tool to allow evaluation of a. 
variety of innovations in instructional technology, particularly those aimed at the 
adaption of instruction to student individual differences.' 

A large, central coirputer (CDC Cyber 73-16) provides data processing capa- 
bility. Data input and output are supported 'by two types of terminals. A 
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management terminal (consisting of a form reader, printer, and minicomputer 
enclosed in a console) is used to read and scor'e test forms, transmit data to the 
central computer, and print out student prescriptions and management reports. 
An Interactive terminal (consisting of a plasma display and keyboard) is used for 
instructor and course author Interaction with AIS data files and will also be used 
for on-line, adaptive testing and CAI. 

The system was designed to provide the functions of diagnosis of student 
performance, forward-going assignment prescriptions, resource allocation an?l 
scheduling, guidance and counseling, information retrieval, report generation, 
and support for evaluation and research. The software to support these functions 
comprises the Adaptive Model and Applications Programs component of the AIS. 
This software was tested and implemented in all four courses during the summer of 
1976/ 

y 

The Adaptive Model contains the component entitled Adapter which assesses 
cognitive and affective student traits and states, and selects instructional 
alternatives that best match the needs of each student at each node of the instruc- 
tional sequence. The individualization afforded by the Adapter is constrained by 
the prerequisite relationships inherent in a course structure and by the availa- 
bility of instructional resources. Thus, a Resource Allocation/Scheduling Model 
within the adaptive model maintains the inventory of available resources, restricts 
the assignment of alternatives to the available resources, supports the scheduling 
of team tasks, and attempts to optimize the utilization of expensive and limited 
training resources. 

Both the Adapter Model and Resource Allocation/Scheduling Model operate on 
predictions as to the amount of time each student will require to complete his 
current assignment and his current course unit. These data, in conjunction with 
predictions o? student scores on each criterion test, are used to provide the 
guidance and counseling function in that each student's performance is continually 
evaluated relative to other students. 

At the time this study was being conducted, the AIS was operating in a more 

rudimentary mode, termed Initial CM I. Although the courses were self-paced, the 

primary means of instruction was limited to programmed textbooks. Instructional 

alternatives were under development, but only a very few had been integrated into 
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the courses. While the main track of instruction did include some instances of 
audio-visual Vessons, the number and variety of these lessons were insufficient 
for the purposes of this study'. While the system provided test scoring and data 
recording capabilities, the sequence 1n which the lessons were studied and the 
assignment of instructional resources were under the control of the classroom 
instructors. Each student- recorded the amount of time he spent in various 
activities, e.g., first-attempt-lesson-study, remedial study, and testing, 
on his individual test answer sheets. These data were recorded by the system, 
but there was no direct measurement of these times. 

4.1.3 Courses Selected - The four courses supported by AIS at the time of the 
study were Inventory Management Specialist, Materiel Facilities Specialist, 
Precision Measuring Equipment' Specialist, and Weapons Mechanic. The analysis to 
be conducted required a large data base (hence a relatively large student 
flow) and a large overlap between the content opf^he current, self-paced course and 
the earlier, conventional course. On the basis of these considerations, the .two 
courses selected for modeling were Inventory Management Specialist (IM) and Weapons 
Mechanic (WM). 

Under conventional instruction, IM was a 210 hour course leading to the 
Air Force Specialty of Inventory Clerk. Course content primarily concerns 
inventory procedures, record keeping, and shipping. The work requires clerical and 
procedure-following skills. The selection criterion for admission is a percentile 
score of 60 on either the General or Administrative Scale of the Armed Services 
Vocational Aptitude B^attery (ASVAB) given all enlisted personnel at the time of 
induction. / 

Under conventional instruction, WM was a 540 hour course leading to a 
specialty in weapons handling. The course teaches the cognitive and psychomotor 
skills required to handle, and load air-to-air and air-to-ground W3apons and to 
troubleshoot and repair related equipment. A key feature of the WM course is its 
employment of team concepts in skill development. The admission criterion Is a 
percentile score of at least 60 on either the Electrical or Mechanical scales of 
the ASVAB. 

Both courses were divided into a number of "blocks" (six in IM; 12 in WM), 
each approximately a week in length. Each block contained from 10 to 30 lessons and 

4-3 

35 

McaaNNBLi, aauGLAS ASTf^a/yAuTtas caMMhiv » cast 



7 



EVALUATING - ,^,,0 
INSTRUCTIONAL SYSTEMS VOLUME II T5 MAY 1977 

an end-of-block test. Students were required to tneet criterion on these block 
tests before proceeding to the nsxt block. Under AIS, lessons are approximately 
one to two^hours in length; each covers two or more instructional Objectives, 
and most conclude with a diayiostic lesson test. Under conventional instruction, 
the lessons had tended to be longeri and, as a rule, a conventional lesson was 
broken up into two or mo^^e AIS lessons. 

\ Of the six blocks 1n the IM course, the first five were analyzed for this 
study. Insufficient student perfprmance data were available from the sixth since 
it had>ecently undergone a major revision. The last four of 12 blocks were 
analyzed in the WM course. The course was being implemented under AIS from back 
to front, and insufficient data were available from the lower numbered blocks. 

/4.2 DEBINITION OF VARIABLES FOR ANALYSIS AND DATA COLLECTION 'PROCEDURES 
4.2.1 Definition of Crite^^ion Variable - Several typfes of time data available 
from AIS were considered for use as the criterion variable to be predicted by the 
fnodel. These included block completion time, first pass block time, lesson 
completion time, first attempt l^son time, first attempt study time, and first 
attempt test time. Of all of these time criteria, block completion time was 
considered to be the most useful criterion and, because of its relationship to 
the descriptive model, the most appropriate. It was, therefore, selected as the 
criterion variable to be predicted, by the model. \ 



wa 



For purposes of developing the predictive model, hov/ever, the number of blocks 
s too small to provide sufficient variability in content. Time prediction 
equations needed to be developed at the lesson 'level . As was noted in the dis- / 
cussiqn of the descriptive model, it could not be assumed that the sum of lesson 
completion times would equal block completion time or even first-pass block time. 
Each pass through the block also includes the time required for the block test 
itself, cumulative administrative overhead which included student "breaks," 
and time lost due to queuing for resources and test scoring. No accurate time 
data were av.ailable' for any of these components. The most appropriate approach, 
therefore, appeared to be one in which the variable predicted would be lesson 
completion 'time. A proportional "lost time" estimate, determined from the dis- 
crepancy between the sum of lesson completion times and block completion time, 

4-4 ^ 

36 

jgpjl^ MCOOfMIVELL OOVCLAS ASTKO/VAUTfCS COMPANY • EAST 



EVALUATING MDC E1570 

INSlbcTIONAL SYSTEMS VOLUME II ^5 MAY 1977 

\ ♦ 

t 

and assumed to contain the time allocated to block remediation, overhead, and 
queuing would then be employed to provide an estimate of block completion time. 

It was, found, however, that lesson completion time was itself extremely 
unreliable. The time required for each attempt on the lesson was recorded on a 
separate test answer sheet with an identifying attempt number. Because of the 
multiple pieces of paper involved and because procedures were not always rigidly 
adhered to In the classrooms, the problems of incomplete and missing data were 
widespread. "The amount of available lesson completion time data which could be 
considered' reliable was inadequate for the modeling study. 

Some consideration Was given to the separate prediction of reported first- 
attempt-study and first-attempt-test time. This approach was discarded, however, 
since there had been a continual problem \n instructing students in how to differen- 
tially record study time and test time due to confusion over self -tests which were 
to be counted as part of study time. The variable finally selected to serve as 
the criterion to be predicted was first-attempt-lesson-time . This variable was 
based on a simple start and stop time recorded on the. student 's answer sheet and 
had been found to be the most reliable of the various time measures. For the 
majority of cases, it was equivalent to lesson completion time. It could be 
related to block time by increasing the magnitude of the proportion of "lost 
time." Viewed in the contextof the Descriptive Model (Figure, 2), the criterion 
measure was the cumulative time (across lessons) attributable to the two components 
entitled First Attempt Study on Lesson i and First Attempt Diagnostic Test on 
Lesson i . ' 

The prediction of block and lesson test scores presented a very different 
problem - that of general izabil ity. It was intended that the predictive model 
to be developed should generalize across a variety of technical training courses. 
While there is no ambiguity concerning the meaning of a clock hour and no vari- 
ation in the meaning of this concept from one course to another, this is not the 
case with regard to test scores. The probability that a given student will obtain 
a passing score on an arbitrary test is at least as much a function of test item 
difficulty as of the student's characteristics and the content being tested. 
Item difficulty is, in turn, primarily a function of the experience and opinions 
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of the developers of the specific instructional materirls and the test. It is 
a safe assumption that the foremost factor Influencing a test developer'sl decisions 
regarding. item difficulty and adequate criterion performance is his past experience 
with the performance of students in this particular course of instruction. 
Thus, while it appeared implausible to generalize across courses with respect 
to expected test score or expected failure rates, it did appear reasonable to expect 
that the level of student performance would remain relatively constant from con- 
ventional to self-paced, individualized instruction. This assumption was made 
explicit in the predictive model being developed. That is, it was assumed that 
the level of wi thin-course student performance, the skill level of the course 
graduates, and the rate of attrition would remain unchanged following the transi- 
tibn from conventional to self-pacedV individualized instruction. The only 
dimension left free to 'vary was, therefore, time. 

4.2.2 Definition of Predictor Variables - The variables selected for investi- 
gation as potential predictors of first-attempt-lesson time will be, discussed in 
terms of the categories of independent variables influencing the instructional 
interaction which were outlined in Figure 2 and discussed in Section 3.0. 

While it would have been plausible to consider bahavioral goals (see Figure 
1 ) as a category of predictor variables, it was considered more appropriate to 
obtain measures of the instructional content itself, the course curriculum. 
Information regarding both classes of factors were available - the Specialty 
Training Standard (STS) reflecting goals and the Plan of Instruction (POI) ref- 
lecting qontent. While the information provided by the STS is more basic and 
less subject to change, the relationship between the two documents should remain 
constant unless the course goals and content are subjected to the type of re- 
analysis called for by an Instructional System Development (ISD) exercise 
(U.S. Arrny Training and Doctrine Command, 1975). The purpose of the predictive 
model is to examine the effects of self-pacing and individualization, jiot the 
effects 6f ISD. It was therefore assumed that content, as reflected in the POI, 
would remain constant during the transistion from conventional to individualized 
instruction. Use of the POI as the reference document for couise content has the 
further advantage of reflecting the influence of local training policy. The specific 
information derived from the POI is described in paragraph 4.2,2.1. 
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As we discussed in paragraph 4.1.2, the Initial CMI mode of operation, which 
was in effect at the^time of this study, did not include computer control led- 
individualization of the type required by the descriptive model, i.e.,, advanced 
placement and assignment of alternative instructional treatments. Therefore, 
no variables pertaining to this factor could be included in the predictor set. 

!n the category of instructional design considerations, it was not feasible 
to include alternative media characteristics as predictors because of the limited 
number of lessons employing media other than text. The extent to which advance, 
embedded, and post-instructional organizers were present in various lessons did 
provide a basis for defining a number of predictors. These are discussed in 
paragraph 4.2.2.2. Predictors in the test characteristics category are discus- 
sed In paragraph 4.2.2.3. In addition, a variety of measures of pre-instruction, 
individual student difference characteristics was investigated. These are 
described in paragraph 4.2.2.4. 

While the quantity and quality of instructional and management system resources 
have a definite effect on the characteristics of the instructional interaction, 
particularly the time required for its completion, there was not sufficient 
variability in this factor among the various AIS courses to permit its use as a 
predictor variable. Consequently, the predictive model assumes an "adequate" 
supply of resources. 

It was recognized that course policy regarding training has a pervasive 
influence on the instructional interaction. It is by nature, however, course- 
specific and relatively intangible. For this initial attempt at model development, 
therefore, it was considered necessary to ignore this factor - to consider its 
effect as contributing to unaccounted for variability except in-so-far as course 
' policy influences were reflected in the instructional content. 

4.2.2.1 Course- Content Variables t Course content characteristics were 
determined from descriptions contained in the plans of instruction (POI) pertaining 
to the five IM and four WM blocks to be analyzed. Each of the Armed Services has 
planning documents containing detailed descriptions of lesson objectives and the 
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distribution of lesson time into categories such as lecture, performance, testing, 
and outside assignment. The variables selected to describe each lesson were based 
on information common to these planning documents, * . 

V ^ 

The content of each AIS lesson was identified in :;.c POI on the basis of 
common objectives. In some cases, the set of objectives covered by a single 
conventional lesson had been transferred directly to an AIS lesson. In general, 
however, the conventional lessons were longer than AIS lessons and contained more 
material. In these cases, two or more artificial conventional lessons were defined, 
each containing the objectives corresponding to a single ^.IS lesson. The total 
instructional time in the conventional lesson and the time allocated to each of 
the instructional methods by the POI were then distributed across the new, arti- 
ficial lessons on the basis of the objectives contained in each new lesson. In a 
similar fashion, if an AIS lesson contained the objectives of two or more con- 
ventional lessons, the times within the POI categories were summed to create an 
artificial conventional lesson. If the depth of content in the conventional 
lesson had been reduced or increased In the corresponding AIS lesson, the POI 
times were increased or reduced, by category, by the corresponding amount. Finally, 
there were some cases in which course content had changed during the AIS implemen- 
tation period, and no conventional instruction POIs were available which included 
the new objectives. When th ^ occurred, the AIS lesson containing the new objec- 
tives was dropped from the analysis. 

These decisions and translations were made by members of the AIS materials 
development teams and content matter specialists who, in many cases, had previously 
taught in the conventional courses. The conclusion of this exercise defined a 
set of AIS lessons in the nine blocks which could be characterized by descriptive 
statements in actual POIs or in POIs constructed artificially, . 

It was obvious that one of the most promising predictors cf first-attempt- 
lesson time under a self-paced system would be the amount of time devoted to that 
lesson under conventional instruction. This variable was defined as total POI 
time. There was reason to expect, however, that predicted first-attempt-lesson 
times (hereafter referred to simply as lesson, time) would be more accurate if 
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separa'te. variables were defined for the various categories of instruction outlined 
by the POI. Four such variables were defined: Lecture/Discussion; Demonstration/ ' 
Perf ormance; Programmed Instruction/Audio-Visual ; and. Outside Assignment" . 

Each AIS lesson contained only material which pertained directly to the stated 
objecti.ves. The "nice to kfiow" information which would usually be included in a 
lecture had been eliminated. Therefore, it ^as assumed that the amount of time 
which the POI allocated to lecture and discussion activities would be shortened 
dramatically under self-pacing. 

Under cbnvehtional instruction, Demons trati on/PeV^formancfi acti vi ti es usually 
involve at least two repetitions of the task to be performed - once by the instruc- 
tor and once by the student. Under self-pacing, the task was performed only once 
by the student under step-by-step guidance 1)y the instructional materials. This 
might be ^slower or faster than was the se under conventional instruction. ' 

Analysis of the objectives, taught c,t least in part through Demonstration/ 
Per^formance activities^ suggested that a further -breakdown might b^ profitable. 
While some of the objectives were primarily psychomotor ir .uture, others appeared 
to be primarily cognitive In nature (e.g., troubleshooting). Therefore, two sub- 
categories wtre defined: Demohstration/'Rerformance-Cognitive ; and Demonstration/ 
Performance-Psychomotor . While the distinct?on dould have been made on a judge- 
mental basis, an alternative approacl\ was employed. Each AIS lesson could, poten- • 
tialiy, contain a performance check, supervisfed by an instructor, as well as* a 
multiple-choice test. If the AIS materials developers had seen f;t to include a 
performance check in the lesson, the POI Lecture/Discussi'on time was categorized 
as Ps ychomotor . Otherwise, the time was categorized as Cognitive . 

,POI instructional time allocated to programmed in^trucjioi. or an audio- 

AT 

v'sual presentation was grouped under a single variable on. the assumption that the 
time required for such activities under self-pacing would not be shortened sub- 
stantially. Finally, Outside Assignment time was treated separately on the assum- 
ption that such activities woulo be less efficient than if they were conducted 
in the classr^pom. ' :/ - - . * ' * 
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In sfimary, six quantitative measures of content were defined: The number 
of minutes allpcated by the POI to (a) Lecture/Discussion, (b) Demonstration/ ' 
Performance-Co.gni t i ve , (c) Demonstration Performance-Psychomotor , (d) Programmed 
instruction/Audio-Visual . (e) Outside Assignmen t; and (f) the "total number of minutes 
allocated to the conventional 'lesson , the sum of the first five measures. 
^* .. " _ , \ • 

Defining measures of the Qualitative characteristics of the content presented 
a substantially different problem. Wh'ile several investigators have offered their 
views regarding the interaction of specific content characterlsti.cs and student 
performance, no one to date has-been able to measure the relationship between 
9 content characteristics and rate of learning in an" operational environment. 
, Tfi current study empL.yed two distinct approaches to the problem of contf.-nt 
classification and conducted a comparative evaluation of the two. The results of 
tbis evaluation are discussed in paragraph 4.3\3.1. . ^ 

The first approach, which will be termed content classification, concentrated 
directly on the lesson content as described by the ^0^1. Three broad categories 
were defined: memory, cognitive, and psychomotor. While few attempts have sought 
to identify such broad categories (most have dealt with more specific facto s In 
well define^ environments, e.g., Guilford, '{1971) , it was thought that this approach 
would have greater utility to instructional pfe^rsonnel given the task of charac- 
terizing course content. • \ 

' \- ■ ■ ■ 

There was a dearth of personnel qualified and available to evaluate each 
lesson in terms of the percent (later translated into minutes) of memory, cognitive, 
and psychometer content involved. For the IM course, two instructors and a third 
party, not a subject matter expert, were asked to make these ratings jointly on the 
29 lessons of interest. Only one subject matter expert was found to make these 
evaluation!; for the 35 lessons of interest in the WM course. For each content 
area^ represented in these lessons, the raters were asked to judge on a scale ot 
one to five how simple or complex the material appeared to be for the t'-ainee popu- 
lation to wnom U was presented. For example, the content of a particular two hour 
lesson might be judged to be 15 percent ,(18 minutes) memory and 85 percent (102 
minutes) psychomotor. The memor^ content might be judged as b^'ing difficult and 
jiven a complexity rating of five. The psychomotor componei.L of the lesson might 
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be judged to be of medium complexity and given a rating of three. Since the lesson 
was devoid of cognitive material , n^ ating was required for that category. The 
precedure of obtaining content ratings for the IM and WM lessons, either by inter- 
judge agreement or by a single judge, respectively, made the determination of 
inter-judge reliability estimates inappropriate. 

Definitions and examples of the three categories and of the complexity scales 
within each category are provided in Table 1. These same d'^ifinitions were given 
to the raters to guide their review of the POIs. These ratings served as input to 
the data base. 

9 

The second approach to categorizing lesson content adopted a more conventional 
method, more oriented toward the terminal behavior described by the objectives 
statdd in the POI. A modification of MerrilTs (1972) classification scheme was 
adopted. MerrilTs categories of discriminated recall, classification, rule using, 
and higher order ruie using were supplemented with the category of psychomotor 
tasks* A binary easy/difficult dimension was then added to each category. Defini- 
tions of the five categories and the bases for the easy/difficult decisions are 
shown in Table 2. 

Two psychologists on the AIS project staff classified each objective listed in 
the relevant portions of the POIs into one of these ten categories. Each lesson 
was then rated on the basis of the proportion of the content attributable to the 
objective(s) in each category. .For example, a particular lesson might contain 
four specific performance objectives. Three of the objectives might be classified 
as Discriminated Recall . Two of these might be judged as Discriminated Recall - 
Easy and the third. Discriminated Recall - Difficult . The fourth objective might 
be categorized as an easy classification task. The POI time allocated to the two 
easy discriminated recall objectives would be assigned to the variable. Discriminated 
Recall - Easy , for this lesson. The time attributable to the other two objectives 
Would be the values of the variables, Discriminated Recall - Difficult and Class - 
ification - Easy . The variables representing the seven remaining categories of 
terminal behavior or perfom)ance, Classification-Difficult through Psychomotor- 
b<f]j^i©aru_vawould be given values of zero. 
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TABLE 1 

CONTENT CLASSIFICATION: DEFINITION 
' AND METHOD OF CALCULATION 



CONTENT AREA 
Memory Related Content 

Memory - Simple/Complex 



Cognitive Content 



Cognitive - Simple/Complex 



Psychomotor Content 



Psychomotor - Simple/Complex 



DESCRIPTION 

That portion of the lesson content re- 
quiring retention of information. 
Measured in terms of the minutes of, 
conventional instruction. 

A five point rating of the simplicity 
or complexity of the memory related 
element in, the lesson. Simple memory 
content (one) includes recognition of 
' familiar terms; complex memory content 
(five) includes recall of many new 
technical terms. 

That portion of the content related* to 
knowledge gathering, understanding 
relationships and principles, and 
problem solving. Measured in terms of 
minutes of conventional instruction. 

A five point rating of the simplicity 
or complexity of the cognitive element 
in the lesson. Simple cognitive 
content (one) involves procedure 
following, labeling, and information 
collecting. Complex .cognitive content 
(five) inv61 es symbol manipulation, 
problem solving, decision making, and 
evaluation. 

That portion of the content related to 
performance or "hands-on" tasks and 
involving^ manual dexterity, eye-hand 
coordination, or gross motor movements 
measured in terms of minutes of 
conventional instruction. 

A five point rating of the simplicity 
or complexity of the performance element. 
Simple psychomotor content (one) includes 
gross motor movements. Complex psycho- 
motor content (five) Involves fine motor 
dexterity, complex eye-hand coordination, 
and precise motor movements. 
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TABLE 2 

BEHAVIOR CLASSIFICATION DEFINITIONS 



BEHAVIOR CATEGORY 
Discriminated Recall -Easy Difficult 



CI assi f i cati on-Easy/Di f f i cul t 



Rule Using-Easy/Difficult 



• Higher Order Rule Using 
Easy/Difficult 



Psychomotor-Easy/Di f f i cul t 



DEFINITION 

Given a symbol, object or event, a 
student responds by providing the ^ 
associated member of the response set. 
If the number of members in the stimulus 
set was large and/or if the members of 
the response set were unfamiliar (e.g., 
new technical terms), the objective was 
judged to be difficult. 

Given a new example from a class of items, 
a stud-^nt responds with class membership. 
The easy/difficult dimension was based 
on the number of classes. 

G^ven a new experience, a student applies 
a rule (a set of operations) to produce 
the appropriate members of the response 
set. Rule Using-Easy was essentially 
following written procedures (e.g., technical 
orders ) . Rule Using-Difficul t was 
following procedures in the absence of 
written guidance. 

Analogous to problem, sol ving activity, the 
retrieval or generation of rules by applying 
a higher order rule to determine problem 
class membership. The easy/difficult 
dimension was based on the number of steps 
in the problem solution. 

The performance of fine or gross motor 
control movements to accomplish a task. 
The easy/difficult dimension was based on 
the familiarity or novelty of the task 
components. 
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The psychologist raters, although familiar with the AIS courses, were by no 
means subject matter experts. Therefore, their ratings were checked and, in a few 
cases, modified, by members of the course materials development teams. This 
procedure was undertaken as an attempt to provide at least face validity for these 
ratinsis* Since independent ratings of the objectives were not obtained, interrater 
• reliability estimates could not be genei .ted. 

4.2.2.2 Instructional Design Variables - As was. previously discussed, AIS, at the 
time of this study, included insufficient examples of lessons taught by media 
other than programmed texts to include alternative me**a as variables 1n the 
predictive model. A similar situation existed with respect to alternative 
instructional treatments designed to compensate for specific deficiencies. Instruc- 
tional design considerations were, therefore, limited to "organizers." Organizers 
were defined as "assists" provided in the instructional material to enhance the 
trainee's achievement of a behavioral objective. 

The most prevalent type of organizers in the AIS materials was the list of 
objectives at the beginning of each lesson. Since every lesson included such a 
list, there was no variability across lessons with respect to this factor; and 
it was, therefore, of no value as a predictor variable. 

Two other types of organizers v^ere present which did vary from one lesson to 
another - embedded questions and self-check items. With the exception of those 
lessons which were strictly performance oriented, embedded questions were scattered 
throughout the textual material. Each consisted of a multiple-choice item, the 
correct answer to which was provided on the last page of the lesson. The majority 
of the AIS lessons, again excluding performance lessons, included a self-check item 
to be completed following study of the lesson but prior to the lesson test. These 
items were also mul tiple -choice. The student marked his response on chemically 
treated paper which then revealed whether or not the response was correct. 

The number of embedded questions and the number- of self-check items were 
recorded for each of the AIS lesson analyzed. If no items of particular type were 
present, a zero was recorded. Although this information was taken directly from 
existing AIS lessons, a tactic which would not be possible if the model were 
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actually being used to predict the characteristics of a seV -paced course yet to be 
constructed, the number of embedded and self-check items was assumed to reflect 
instructional design policies v^hich could and should be established prior to actual 
materials development. 

4.2.2.3 Test Characteristi cs - Consideration of test characteristics was limited 
to the lesson tests themselves. Due to the inclusion of new material in the AIS 
lessons, there were no examples of complete blocks^ Thus, all of the block tests 
included some items from lessons which were not sampled for this study. 

Three test characteristics were measured for each lesson test. For each lesson 
which included a multiple-choice test, the number of test items and the test 
criterion (expressed as a percentage) were recorded. For lessons without multiple,- 
choice^ests, a zero was reqorded for both number of items and criterion. For each 
lesson whicfr-in^luded a performance test, a situation in which the student was to 
perform a task to the instructor's satisfaction, the number of specific checkpoints 
in the performance task was recorded. Again, a zero was recorded for this variable 
if the lesson did not include a performance test. 

4.2.2.4 Student Characteristics - The variables chosen as student characteristics 
were determined o :he basis of (a) the ease with which they could be obtained in a 
variety of military training environments, (b) how representative they were of a 
wide variety of individual differences, both cognitive and affective, end (c) their 
known relationship to the prediction of AIS training time criteria in different 
types of training courses. Thus, the resulting student characteristic variables 
include the four primary composite scores from the Armed Services Vocational 
Aptitude Battery (ASVAB), sex, and selected affective and cognitive variables 
which were common to the AIS pre-assessment batteries of both the IM and WM 
courses. 

A complete description of each of the selected student characteristic variables 
is presented in Table 3. The variables are organized in th^ee categories: (a) Armed 
Services Vocational Aptitude Battery (ASVAB) Variables and sex; (b) Cognitive AIS 
Pre-Assessment Battery Variables; and (c) Affective AIS Pre-Assessment Battery 
Variables. Wherever applicable, the references for a particular measure of student 
characteristic variables are cited. 
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TABLE 3 

STUDENT CHARACTERISTIC VARIABLES 



VARIABLE CATEGORY 



VARIABLE NAMES 



DEFINITION 



I. ASVAB Composites 
and Sex 



II. ^.Cognitive AIS 
Pre-Assessment 
Variables 



III. Affective AIS 
Pre-Assessment 
Variables: 



ASVAB Scores: 
General Ability 

Electrical Ability 

Administrative Ability 

Mechanical Ability 

Sex 



Reading Vocabulary 
Test - Total Scores 



Reading Skills Scales 
1 and 2 



Concealed Figures Test 



Pre-Gourse State 
Curiosity Scale 



Pre-Course State 
Anxiety Scale 



A standard vocational aptitude 
battery, administered to all 
enlisted inductees, which 
contains four primary composite 
scores for general, electrical, 
administrative, and mechanical 
abilities, respectively. 

An identifier' of individual 
differences in training times 
attributable to sex differences,, 
where males = 1, and females = 0. 

A 45-item measure of students' 
ability to recognize the defini- 
tion of terms frequently used in 
7\ir Force documents (Diegnan, 1973)/ 

Course-specific measures of 
students' reading comprehension 
and speed on materials extracted 
from IM and WM technical .manuals. 
Scales 1 and 2, consisting of 10^ 
items each, differ only in terms/ 
of difficulty, with longer pas- 
sages and more rigid time limits 
imposed on Seal 6 2^(McCombs, 1^74). 

A 20-item measure of students 
ability to make perceptual disl- 
tinctions by recognizing which 
of five simple geometrical fi^gures 
is embedded within a complex^ 
pattern (Diegnan, 1974). 

A 20-item self-report measure 
of students' anticipated feelings 
of interest in learning the IM 
or WM course materials (Leherissey- 
McCombs, 1971). 

A 20-item self-report measure of 
students' anticipated feelings of 
tension or apprehension about 
learning the IM or WM course 
materials (Spielberger, Gorsuch, 
& Lushene, 1970). 
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TABLE 3 (cont.) 
STUDENT CHARACTERISTIC VARIABLES 



VARIABLE CATEGORY 

IIL Affective AIS' 
Pre-Assessment 
Variables (cont.) 



VARIABLE NAMES 

Test Anxiety 
Questionnaire 



Internal -External 
Scale 



General Media 
Preference and 
Instructional 
Experience Subscales: 
Audio Preference 

Visual Preference 

Printed Preference 

Experience with Con- 
ventional Institution 

Experience with Self- 
Paced Instruction 



DEFINITION 



A 16-item self* report measure of 
students* tendencies to become 
anxious when taking ability or 
achievements tests (Sarason, 1958). 

A 29-item self-report measure 
of students* tendencies to 
feel in control of events 
versus controlled by external 
events (Rotter, 1966). 

9 

A 15-item self-report bcale, 
consisting of five subscales 
which measure students' general 
preference for audio versus 
printed maten*als, as well as 
their reported degree of 
experience with conventional 
versus self-paced instruction 
(McCombs, 1974) 
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The primary rationale for the selection of the four ASVAB variables and sex 
was that these variables are easily obtained on enlisted trainees in all branches 
of the military lervices. In addition, many years of systematic research with the 
ASVAB battery has shown it to be a highly reliable and valid measure of individual 
differences in vocational aptitudes. A number of research studies, as well as our 
own experience with AIS data, have shown sex to be related to individual differences 
in training times. 

Within the Cognitive AIS Pre-Assessment Battery Variables category, three 
reading ability measures and one perceptual reasoning ability measure were chosen. 
The reading variables have been shown to be highly predictive of independent 
variance in training time criteria in each of the four AIS courses. In addition, 
it was felt that, even though these particular reading measures might not be 
available in other military applications, similar reading vocabulary, comprehensive, 
or speed measures were likely to be available. The perceptual reasoning variable, 
defined by scores on the Concealed Figures Test, was selected on the basis that it 
was available for both IM and WM students, and it represented a perceptual/reasoning 
ability that would be generally related to student performance in a variety of 
technical specialties. At the time this variable was selected, preliminary AIS. data 
indicated that it was moderately related to training times on some of the IM course 
materials. 

. The particular measures chosen with the Affective AIS Pre-Assessment Variable 
category were chosen on the basis of (a) their known relationship to AIS training 
time criteria; (b) the wide range of student motivations, attitudes, and learning 
style preferences they represented; and (c) their availability as common predictor 
variables in both the IM and WM courses. For example, state curiosity, state anx- 
iety, and test anxiety have all been shown to be highly predictive of AIS training 
time criteria (McCombs and Siering, 1976). Although at the time of this study not 
much AIS data were available of the Internal -External Scale, it was judged to be a 
student characteristic variable that would be expected to be highly related to 
student training times in selfj-paced instructional environments. Similarly, 
students' preferences for audio versus visual printed instructional modes of learn- 
ing, and their experience with self-paced versus conventional instruction were also 
judged to be important affective predictor variables in self-paced instructional 
environments. 
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4.2.3 Creation of Files for Data Analysis - Creating the data files was a lengthy 
process which involved merging three distinct classes of data: 

0 Student data available from the AIS data analysis system: Course; Social 

Security Number (u^-.ed by AIS as an ID); sex; preassessment test data; 

block and lesson number; module number (i.e., instructional alternative 

type); first-attempt-lesson time; and first-pass-block time 
0 Supplementary student data: The ASVAB General, Electrical, Administrative, 

and Mechanical scores 
0 Lesson characteristic data: the POI derived quantitative measures ; content 

classification measures; behavior classification measures; instructional 

design measures; and test characteristic measures. 

Student data were extracted from AIS files y^epresenting the nine blocks of 
the two courses using the AIS Data Extraction Program (DEP). It was necessary to 



extract data from sc-parate and distinct time periods in ov^der to avoid time 
periods in which the courses were undergoing substantive changes. The product of 
this activity was nineteen separate files. 

ASVAB scores were not normally recorded in the AIS files since they were 
not usually available early enough in the course to be of value as predictors. 
For purposes of this study, however, ASVAB scores were obtained from Air Training 
Command's Training Management Information System (TRAMIS) for as many of the students 
as possible represented in the DEP files described above. The scores were obtained 
on cards, input into the computer, and an additional file was constructed. 

A total of 64 lessons was sampled. Twenty-nine of these were from the IM 
course; six from block one; six from block two, six from block three, four from 
block four; and seven from block five. The remaining 35 were drawn from the WM 
course: nine from block nine; eight from block ten; ten from block eleven; and 
eight from block twelve. Lesson characteristic data were punched on cards, input, 
and a file was created for each of the nine blocks. 
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A special purpose program was written to merge files across categories. Each 
run of this program produced a file containing all relevant data for each of the 
nineteen DEP files previously created. These nineteen files were then merged into 
a single tape file containing approximately 13,700 records where each record 
represented a particular student's first attempt on a lesson of known characteristics. 
That is, each record contained the relevant characteristics of both the student and 
the lesson as well as the time required for his first attempt on the lesson. 

The magnitude of this file made its retention and manipulation on a disk 
impractical. The study required two comparable samples, one for regression 
equation generation and a second for equation validation. Therefore, the file was 
immediately split into two samples on the basis of whether the final digit of a 
trainee's Social Security Number was odd or even. After excess records had been 
deleted from the larger sample, each sample contained 6849 records. 

Missing and incorrect data were rampant throughout the developing AIS data 
base. Typical problems included preassessment forms which were incorrectly read 
by the system's optical scanner, and trainees who were enrolled without completing 
any of the preassessment tests. ASVAB. scores obtained from TRAMIS were often 
incomplete or missing entirely. First-attempt-lesson-times contained a small, but 
persistent number of widely deviant values. The primary cause of these outliers 
was miscoding, e.g., if a student worked on a lesson during two consecutive days, 
the number of days was often mismarked on the test answer sheet. The measure of 
first-pass-block-time was also error prone. It was defined as the elapsed time 
from starting the first lesson in the block until the block test was completed. 
Errors such as mismarked dates often made this value fallacious. 

To reduce these errors as much as possible, sets of constraints were established 
for these variables. Preassessment scores were recoded as missing values if they 
were actually missing or exceeded the known limits of the tes.t scores. All ASVAB 
. scores, if present, were determined to be reasonable. First-attempt-lesson-times were 
limited to a range of fifteen to three-hundred ininutes. This eliminated approxi- 
. mately five percent of the data from either end of the distribution. Block times 
were excluded if the reported time was not greater than the sum of the first-attempt- 
lesson-times or ir the reported time fell outside the range established as 
reasonable by prior experience with AIS. 
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The regression analyses was conducted using the (SPSS) Statistical Package 
for the Social Sciences (Nie, Hull, Jenkins, Steinbrenner, and Br^nt, 1975). SPSS 
offers two options with regard to missing data: "pair-wise deletion" which 
includes all of the data points available for a particular variable in the analysis, 
and "list or case-wise deletion"-which excludes any case from consideration if that 
case has one or more missing data points in any of the var»aoles included in the 
analysis. Although the former option is less expensive in terms of lost data,, 
it tends to result in fallaciously high correlation coefficients. Therefore, the 
case-wise deletion option was selected. This reduced Sample One, the data base 
to be used for regression equation generation, to 2974 cases, each having a complete 
set of data points for each variable. It should be noted that this figure does not 
represent 2974 different students. Varying numbers of observations were generated 
by individual students. For example, data generated by a particular IM student might 
be present for the data base for three lessons in Block One and two lessons in 
Block Two. The implications of these repeated measures are discussed in paragraph 
4.3.1 below. 

A summarizing list of the variables investigated, their means and standard 
deviations (based on the final 2974 cases), and their correlation with the criterion 
variable is shown in Table 4. Given the size of the sample, first order correlations 
equal to or greater than R = +.05 are significant (p < .05). 

4.3 STATISTICAL ANALYSIS AND RESULTS 

4.3.1 Description of Statistical Methods - The uata were analyzed using several 
statistical methods, primarily programs from the Statistical Package for the 
Social Sciences (Nie at al 1975). Available SPSS regression program options were 
used to constrain data values to reasonable limits and to screen out missing data. ^ 
The two primary methods employed in obtaining the final prediction equation were 
linear model comparisons (Ward and Jennings, 1973) and stepwise multiple regres- 
stion analysis. 

In linear model comparisons, two linear models (i.e., regression equations) 
are compared in terms of the reduction of the multiple correlation coefficient 
squared (R ). The model with the larger number of predictors is referred -to as 
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TABLE 4 

MEANS, STANDARD DEVIATIONS, AND FIRST ORDER CORRELATIONS OF 
PREDICTOR AND CRITERION VARIABLES 



VARIABLES 



COURSE CONTENT VARIABLES 
Quantitative Measures (from POI). 

Lecture/Discussion Minutes 
Demonstration/Performance Minutes - 

Cognitive 
Demonstration/Performance Minutes - 

Psychomotor 
Programmed Instruction/AV Minutes 
Outside Assignment Minutes 

C ontent Classification Measures 

Memory Content Minutes 
Simple/Comple:( Memory Rating (1-5) 
Cognitive Content Minutes 
Simple/Complex Cognitive Rating (1-5)- 
Psychomotor Content Minutes 
Simple/Complex Psychomotor Rating (1-5) 



MEAN 



STD. 
DEV. 



(All 



Behavior Classification Measures 
Values in Minutes ) 
Discriminated Recall - Easy 
Discriminated Recall - 
Difficult 

Classification - Easy" 
Classification - Difficult 
Rule Using - Easy 
Rule Using - Difficult 
Higher Order R.U. - Easy 
Higher Order R.U. - Difficult 
Psychomotor - Easy 
Psychomotor - Difficult 

INSTRUCTIONAL DESIGN VARIABLES 

No. of Embedded Questions 
No. of Self-Check Items 

TEST CHARACTERISTIC VARIABLES 



No. 6f Lesson Mastery Test Items 

Master Test Criterion (Percent) 

No. of Performance Test Check Points 



68.86 
2.61 
145.67 
2.81 
20.95 
1.23 



86.13 



71.99 
0.74 
147.37 
1.05 
53.13 
1.90 



126.06 



R WITH - 
CRITERION" 



89.54 


97.58 


.489 


55.70 


64.45 


.488 


27.91 


63.19. 


-.294 


8.43 


38.67 


. 047 


51 .27 


127.10 


.029 



.321 

.201 
.329 

r.015 

-.084 
-.239 



,102 



25.88 58.08 .049 
1.02' 8.72 -.079 

(no instances of this category), 
86.56 111.85 .368 

4.18 19.43 .138 

13.13 37.00 .349 

1.57 11.82 -.Q99 

14.30 36.23 -.122 

(no instances of this category} 



16.27 
2.74 



9.38 
57.70 
8.26 



1 5.18 
4.54 



6.69 
29.75 
19.04 



590 
149 



.478 
.288 
.279 
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— - . .TABLE 4 

MEANS, STANDARD DEVIATIONS AND FIRST ORDER CORRELATIONS OF 
PREDICTOR AND CRITERION VARIABLES 

(Cont'd) 



VARIABLES 



MEAN 



STD. 
DEV. 



STUDENT CHARACTERISTIC VARIABLES 

Sex (0 = Female, 1 = Male) 0.87 

■ ASVAB SCO 

Generr.l .. 68.85 

Adnyni strati ve 59.6, 

Mechanical ' 58.97 

Electrical 65.98 

Cognitive AIS Preassessment S cores 

Reading Vocabulary . 2:^.37 

Reading Skills Scale 1 5.38 

Reading SkHls Scale 2 5.; 8 

Concealed/Figures . 7.96 

Affective AIS Preassessment Scores 

State Curiosity ' 65.76 

State Anxiety 38.89 

Test Anxiety 30.11 

Internal -External 14.43 

. Audio Preference 1I.O9 

Visual Preference 7.89 

Printed Preference 4.39 

Conventional Instruction Experience 5.94 

Self-Paced Instruction Experience 5.45 

CRITERION* VARIABLE 

Fi rst -Attempt -Lesson Time ■ 84.90 



0.33 



13.30 
18.47 
25.88 
1 6.65 



6.80 
1.80 
2J8 
'3.60 



7.77 
8.3P 
7. , 
^.88 
04 
22 
44 
64 
35 



R WPH 
CRITlRION 



283 



,089 
,149 
.384 
,252 



,041 
,160 
,218 
,042 



.049 
.084 
.182 
.021 
.007 
.147 
,069 
.012 
,095 



64.74 
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the full model, md the model with the smaller number of predictors is called the 

2 \ 
restricted model, '''hs reduction in R is F distributed with degrees of freedom ' 

df^j/df^ according to the following equation. 

^ ^ - R,^) -/ (df)^ 
(1 - r/) / (df)2 

2 2 

where R . = R of the full model, 
2 2 ' 

/ = R of the restricted model, 

(df)"! ~ the difference between the number of linearly 

independent predictor vectors of the full (n^) and 

restricted (n^) models, and 
{df)2 = N - n^ where N is the total number of cases. 

For the linear model comparisons in this study, SPSS regression program options 
were uSed to delete predictor variables and to generate composite predictors. Ihe 
R Values were obtained from the SPSS program, and the F values were obtained from 
a FORTRAN program produced by the AIS project staff. Stepwise multiple regression 
analysis is simply multiple regression analysis perfornied by adding, step by step, 
the predictor variabler which^i .ds th nighest F vaiue for the increase in R^ 
according to the above equatioi^. The SPSS multiple regression programs include a 
stepwise option which was employed where appropriate for this study. 

The models employed in this series of linear model comparisons did not control 
for the effect of repeiated measures on the lesson variables. There are several 
reasons why this was not done. First, one controls for the effect of a particular 
variable, e.g., subjects, to increase the precision of an F-test between, two models. 
The very large this study provided sufficient precision so that additional 
control over ext'^aneous error variance was not necessary. In general, the F-tests 
resulting from model comparisons were highly significant. Secor^. the effects 
of the repeated measures were curtailed by the fact that the repeated observations 
for a given subject were for different lessons (different learning tasks) rather 
than on the same lesson and that subjects with repeated measures varied with 
regard to the sets of lessons on which thp measures were repeated. Third, to con- 
trol for the repeated measures would have required the generation of a bina\v 
vector to represent each individual student. This would have far exceeded the 
capacity of the SPSS program. Finally, it must be remembered that the purpose 
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of the resultant regression model was to predict first-attempt-lesson-time given 
only the characteristics of the lesson and of a generally defined student population. 
Thus, the user of the prediction equation would have no means by which to identify 
individual students, 

' e 

Cross validation of the final regression equation model also employed the 
SPSS regression program. SPSS regression program options were used to define a^ 
\ i'^ble, predicted first-attempt-lesson-time, which was the result of the applica- 

1 of the final regression equation to data from Sample Two. The values of this 
variable were than correlated with actual Sample Two, first-attempt-lesson-times. 

4.3.? Elimination of Two Variables Highly Correlated with the Criterion - Early 
analyses indicated the presence of two variables which were highly correlated with 
the criterion of first-attempt-lesson-time. These were the instructional design 
variables. N umber of Embedded Questions (r = .590), and the test characteristic 
variable, Number of Lesson Test Items (r = .478). Since these variables were also 
highly correlated with a number of other variables, they tended to mask the 
relevance of these potential predictors and accounted for more of the, variabil ity 
than was logically jDstified. 

It will be recalled that the numbers of Embedded Questions and Lesson Test Items 
were taken directl/ from existing AIS materials. This action was justified on the 
assumption that the numbers of items reflected instructional design decisions 
which could be made prior to the development of self-paced materials. Unfortunately 
for this study, AIS material developers emnloyed a'cor.sistent policy with regard 
to the number of embedded and lesson test questions throughout their instructional 
development effort in both of the courses sampled. Thus, almost all of the 
variability in these two variables could be traced directly to the amount of subject 
matter content In each lesson an)(^, hence, its average first-attempt-time. Since the 
current study was basically a suryey,,and direct manipulation of the various pre- 
dictor variables was not feasible,, some problem^ of this type had to be expected. 
Despite the resultant loss of predictability, it was imperative that these two 
variables be deleted from the predictor set, in order to obtain a realistic assess- 
ment of the relevance of the remaining variables. 
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Two linear model comparisons were made in eliminating these two variables. The 

full regression model, prior to elimination of the variables, contained 34 

predictors - all of those, other than the behavior classification measures, are 

shown in Table 4. The multiple correlation coefficient obtained was .68599, R = 

.47058. Elimination of the Embedded Questions variable reduced proportion of the 

2 

variance accounted for to R = .42525. A test for the signif icanc^e of the reduction 
In resulted in an F (1 , 2939) = 251.643, £ < .001. When the variable Number of 
Lesion Test Items was removed, R^ was further reduced to .40413, £ (1, 2940) = 
108.034, £ < .001. 

4. 3. ,3 Reduction of Predictor Set Through Linear Models Comparisions - A series of 
linear model comparisons was made in order systematically to obtain a more parsi- 
monious set of predictor variables. Although the specific questions of interest 
could have been asked by isolating the relevant variables, it was considered more 
appropriate that the comparisons be made in the context of the complete set of 
remaining predictors. In all of the comparisons to be discussed, emphasis was 
placed on cost-effectiveness rather than on statistical significance. Due to the 
magnitude of the data base, almost all of the linear model comparisons resulted in 
significant differences. In general, however, the increased cost of obtaining 
data required by the additional, variables in the full model did not appear to be 
justified by the iacrement in predictability provi ! by the full model. Therefore, 

the decision rule adopted was to select the restricted model unless the reduction 

2 ' 
in R approached or exceeded .01000". , 

4.3.3.1 Course Content and BehavioV^^lassif ication Measures - An obvious first 

step in reducing the variable set was to select one of the two approaches for 

classifying course content. Prior to comparing the two approaches, however, the 

exten^t to which the number of variables in each set could be reduced was investi- 
gated. The content classification scheme was attacked first. 

The initial full model. Model 1, contained the 32 variables remaining after 
Number of Embedded Questions and Number of Lesson Test Items had been deleted and 
excluding the benavior classification Variables^ The multiple correlation 
generated by Model 1 was .63572, R^ = .40413. 
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The first question asked concerned whether the simplicity/complexity dimension 
employed by this approach added substantially to the scheme's predictive utility. 
A restricted model, Model 2, was defined from Model 1 by deleting the following 
three rating scale variables: Memo^7 - Simple/Complex ; Cognitive - Simple/Complex ; 
and Psychomotor - Simple/Complex . Model 2 resulted in an R of .38395, a reduction 
from Model 1 of .02018, F (3, 2941) = 33.200, £ < .001. This was considered to be 
too vjreat a loss in predictability for the cost of obtaining the simple/complex 
ratings, and the decision was made to retain the rating scales, i.e.. Model 1. 

Attention was then directed to the behavior classification approach. A full 
model. Model 3, was defined which consisted of 34 predictor variables. Model 3 
differed from Model 1 in that the eight behavior classification measures 
were substituted for the six content classification measures. As was noted in 
Table 4, the lessons sampled provided data for only eight of the ten behavior 
classification measures. No instances of Difficult Classification or Diff icul t 
Psychomotor tasks were encountered. 

Model 3 resulted in an P.^ of .36057 as compared with an of .40413 for 
Model 1. Thus, it was immediately obvious that the content classification approach 
was preferable in terms of level of predictability. The proposed questions of 
interest regarding the behavior classification approach were conducted anyway. 

. • The first question asked was similar to that asked regarding the content 
classification scheme - could the easy/difficult dimension be dropped without 
substc.ntive loss in predictability? A restricted model. Model 4, was defined in 
which the eight available behavior classification predictor variables were reduced 
to five. The values of Discriminated Recall -Easy and -Difficult , were added 
together to produce a simple Discriminated Recall variable. The same procedure 

was repeated with respect to Rule Using-Easy and -Difficult and H igher Order Rule 

' 2 

Usinq-Easy and -Difficult. Model 4 resulted in an R of .35451, a reduction from 
2 

Model 3 of .00606, £ (3, 2939) = 9.285, ^ < .001. Since the reduction in R was 
substantially less than 0.01, deletion of this dimension would appear justified in 
view'of the difficulty of obtaining the easy/difficult ratings and their probable 
unreliabil ity. 
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A second question was concerned with the utility of the behavior category 
Higher Order Rule Using . Relatively few of the lesson samples contained examples 
of this type of behavior. It was thought that this might well be a typical 
circumstance in military technical training where even most troubleshooting tasks 
have largely been reduced to a procedure-following activity. Therefore, a further 
restricted model, Model 5, was defined in which the values of the variables Higher 
Order Rule Using and Rule Using were added together to produce a single Rule Using 
variable. Model 5 resulted in an R of : 35443, a reduction from Model 4 of only 
.00008, £ (1, '2942) = 0.365, p = .546. This was obviously not a sufficiently 
substantive loss in predictability to justify generation of the two separate 
categories Rule Using and Higher Order Rule Using . 

As was mentioned above, the full behavior classification model (Model 3) 

2 

resulted in a smaller R than did the final content classification model (Model 1). 
This was despite the fact that Model. X^contained fewer predictor variables (32) 
than did Model 3 (34). An F test was therefore inappropriate as well as unnecessary, 
and Model 1 was selected. That is, the six content classif i^aj:ion variables were 
retained as predictors, and the behavior classification variables^>er.e deleted 
from the predictor set. ^'^'^'^^ 



Of the two approaches, the content classification scheme was also considered 



were generated by course materials development and instructional personnel, the 
type of personnel who would be most readily available to rate the content of a 
course being considered for salf-pacing. The more esoteric behavior classification 



concepts and terminology. It can be assumed that more extensive rater training 
would be required if course personnel were required to produce the behavior 
ratings. 

4.3.3.2 Quantitative Course Content Measures - It will be recalled that the time 
allocated to the conventional lesson by the POI was broken down into five categories: 
Lecture/Pi scjss ion ; Demons tration/Performance-Coqni tive ; Demonstrati on/Performance- 
Psychomotor ; Programmed Instruction/AV ; and Outside Assignment . The question of 
interest concerned the necessity of these five categories. The most likely 




more desirable on practical grounds. It will be recalled that the content ratings 



scheme was'' conducted by instructional psychologists familiar with the behavioral 
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candidate for reduction appeared to be the Cognitive and Psychomotor subsets of 
Demonstration/Performance 'time. The model resulting from the course content 
comparisons, Model 1, \vas employed as the full model. The restricted model, 
Model 6, differed from Model 1 in that it included only a single Demonstration/ 
Performance variable, and was generated by summing the values of the Demonstration/ 
Performance-Cognitive and -Psychomotor variables in Model 1. Model 6 resulted in 
R of. 38800, a reduction from Model 1 of\oi613, F (1 , 294) = 79.612, p < .001. 
The cost of generating these two sub-categories was considered justified by the 
increased predictability. Model 1 was again retained. 

The overall utility of breaking total conventional lesson time down into five 
components was of some interest. To investigate this question. Model 1 was again 
employed as the full model, and a restricted model. Model 7, was defined which 
contained a single quantitative measure of course content - the total number of 
minutes allocated to the lesson by the POI. This variable v/as generated by summing 
the values of the five component times. Model 7 resulted in an of .29748, a 
reduction from Model 1 of .10665, F (1, 2941) = 131.597, £ < .001. There would 
appear to be no question but that the use of "the individual component times is 
well justified. 

4.3.3.3 Student Characteristic Measures - The final set of comparisons connected 
the utility of the various AIS pre-assessment measures. While it can be assumed 
that ASVAB scores will generally be available, the cost of obtaining data comparable 
to the pre-assessment measures could only be justified if they made a substantial 
contribution to the predictive model's utility. Of the two categories of pre« 
assessment measures, the affective measures were considered the most promising. 
^It^was reasoned that the cognitive measures would probably not add much to a pre- 
dictor set which already contained ASVAB scores. A restricted model. Model 8, 
was defined from which the four cognitive pre-assessment measures were deleted. 
Model 1 was employed as the full model. The restricted model resulted in an R 
of .40230, a reduction from Model 1 of only .00183. This difference was not 
significant (F (4, 2941) = 2.258, £ = .061). The restricted model, Model 8, was 
t' ^refore selected. 
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Finally, the effect of deleting the affective variables was evaluated. Model 
8 was employed as the full model, and a restricted model. Model 9, was defined 
which contained no pre-assessment variables. Model 9 generated an R of .3957G, a 
reduction from Model 8 of -00652, F (9, 2945) = 3,570, p < ,001, Although the 
difference was significant, the increased predictability contributed by the affec- 
tive pre-assessment measures was not considered to justify their cost for this 
application. Therefore, Model 9 was retained as the final predictor set. 

The limited utility of the preassessment variables for this study should not 
be generalized to their function as predictors in an individualized instructional 
system. The preassessment variables selected were those common to both the IM 
and WM courses; however, the course content levels sampled by this study were much 
more heterogeneous than would generally be expected in a single, technical training 
course. Course-specific, individual difference measures of the type represented 
in the AIS preassessment battery could well provide the finer differentiation 
required when dealing with students in a single course. More importantly, it must 
be remembered that the initial CMI mode of AIS operation from which this Study's 
data were drawn did not include individualized assignment of instructional alterna- 
tives. If a proposed self-paced course were to include alternative instructional 
treatments assigned on an individual basis, measures such as those in the pre- 
asses.^.ient battery could well make a substantive contribution in a predictive model 

4,3,4 Resultant Regression Model to Predict First-Attempt-Lesson-Time - The 19 
variables contained in Model 9 were submitted to the SPSS stepwise Multiple Linear 
Regression Analysis program. On the basis of this analysis, 14 of the 19 variables 
were selected for the finaT regression equation. The cutoff was made at the point 
at which no one of the remaining variables significantly (£ < .01) increased the 
overall multiple correlation. The final 14 variables, their coefficients to five 
significant digits, and the multiple R and R^ at each step are shown in lable 5 in 
their order of entry into the regression equation. 

, The final regression equation produced a multiple R of .62832, thus accounting 
for 39.48 percent of the variance in first-attempt-lesson-time for Sanple 1. All 
of the five quantitative course content measures entered the equation. As would be 
expected, the time required to teach the lesson in the conventional course was 
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TABLE 5 

MULTIPLE LINEAR REGRESSION EQUATION 
PREDICTING 
FIRST ATTEMPT LESSON TIME 



VARIABLE 


COEFFICIENT 


BETA WEIGHT 


MULTIPLE R 


R2 


Constant 


58.136 








Lecture/ Discuss ion Minutes 


0.11689 


0.17618 


.4886 


.2388 


Demonstration/Pei rormance 
Minutes-Cognitive 


0,25768 


0.25651 


.5608 


.3145 


ASVAB Mechanical Score 


-0.26054 


-0.10415 


.5764 


.3322 


Memory Content Minutes 


-0.17944 


-0.19951 


.5883 


.3461 


Si rnol p/Cornol px Memorv 
Rating 


14.928 


0.17171 


.6005 


.3606 


No. of Self Check Items 


-0.98747^ 


-0.069196 


.6047. 


.3657 


Psychomotor Content Minutp<; 


-0 20393 




. uu^o 




ProQrammed Instruction/AV 
Minutes 


0.27739 


0.16567 


.6203 


.3848 


Sex 


-16.202 


-0.083742 


.6238 


.3891 


Demonstration/Performance 
Minutes -Psychomotor 


-0.088128 


-0.086009 


.6255 ' 


.3913 


ASVAB Administrative Score 


-0.10187 


•0.029058 


.6261 


.3919 


Simp^e/Com^J^ex Psychomotor 
Rating 


1 . 9439 


0.056918 


.6?' 


.3927 


Cognitive Content Minutes 


0.15746 


0.35841 




.3934 


Outside Assignment Minutes 


-0.14308 


-0.28089 


.6283 


.394PJ 
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Strongly related to the self-paced lesson completion time. Five of the six content 
classification measures were included^. It is interesting to note that the measures 
Memory Content Minutes and Psychomotor Content Minutes were inversely correlated 
with the criterion, i.e., the greater the amount of this type of content, the 
shorter the first-attempt-lessbn-time. This implies that substantial time reduc- 
tions can be realized through self-pacing in these areas. The one remaining 
instructional design variable. Number of Self-Check Items , also entered, with a neg-*" 
ative coefficient. This would imply that the time required for^such post-instruc- 
tional organizers is more than justified. Neither of the tv/o remaining test 
characteristic variables. Lesson Test Criterion or Number of Performance Te:t Check 
Points was fomd to contribute significantly to the multiple regression equation. 
With regard to student characteristic variables, sex was found to te relevant, 
entering with a negative coefficient: implying that, for the material sampled, males 
tend to progress more quickly than do females. Finally, the ASVAB Mechanical and 
Administrative scores entered with, as would be expected, negative coefficients. 
Given the presence of those two scores, ASVAB General was not found to contribute 
significantly to the equation. 

4.3.5 Test of the Model's Validity and Generality - The generality of the 
regression equation model was evaluated by two means, cross validation and a test 
of course independence. The same constraints which had been applied to Sample One 
were applied to the 6849 cases in Sample Two, v/hich, along with the casewise 
deletion of mission data, reduced the sample size to 2830 cases. The Sample Twc 
means and standard deviations of the 14 predictor variables, and the criterion 
variable are shown in Table 6. The comparable Sample One values are also shown 
for the sake of comparison. 

For the cross validation, the final regression equation, shown in Table 5, was 
applied to the Sample Two data. The resultant distribution of predicted first- 
attempt -lessoji -times had a mean of 83.91 and a standard deviation of 39.23, The 

correlation between the predicted and actual first-attempt-lesson-times was .5843, 
2 

R = .3414 with a standard error of estimate of 50.62 minutes. This correlation 
was significantly greater than .000 (£ (1 , 2828) = 1465.840, < .001). Since the 
original multiple correlaticn coefficient was .5283, the shrinkage in cross 
validation amounted to .04400. In terms of R , the loss in percentage of the 
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TABLE 6 

MEANS AND STANDARD DEVIATIONS OF RELEVANT VARIABLES FROM 
SAMPLE USED TO GENERATE REGRESSION EQUATION (SAMPLE ONE) 
AND SAMPLE USED FOR CROSS VALIDATION (SAMPLE TWO) 



VARIABJ-E 


SAMPLE ONE 


SAMPLE TWO 


MEAN 


STD. DEV. 


MEAN 


CTn nr\/ 

blD. DEV. 


fOURSF rDNTFNT VflDTflRl F<; 










fluri n t i t t "i VP Mp;*QiirPQ 

v^uaiiu 1 Uuu ivc rlCu^Ui Co 










1 Prturp/niQniQQinn Mimifpc 

ucv* w>u 1 u loGUoo lUil rl IllUUCb 


on CA 

o9. 04 


C\1 CO 

97. b8 


oo . UU 


yo . 1 y 


npmnn t inn /Ppr"fn rm^inrp 

Minutes-Cognitive 


55.70 


64:45 


53.74 


63.10 


Demonstration/Performance 
Minutes - Psychomotor 


27.91 


63.19 


27.75 


62.42 


Programmed Instruction/AV 
Minutes 


8.43 


38.67 


8.42 \ 


38.39 


Outside Assignment Minutes 


51 .27 


127.10 


49.65 \ 


120.21 


Content Classification Measures 










Memory Content Minutes 


68.86 


71 .99 


68.57 


! 72.33 


Simple/Complex Memory Rating 


2.61 


0.74 


2.62 


0.75 - 


Cognitive Content Minutes 


145.67 


147.37 


140.26 


142.48 


Psychomotor Content Minutes 


20.95 


53.13 


20.83 


51.18 


Simple/Complex Psychomotor 

pa 1" "i nn 


1 .23 


1 .90 


1 .25 


1.93 


Instructional Design Variables 










No. of Self -Check Items 


2.74 


4.54 


2.85 


4.65 


STUDENT CHARACTERISTIC 










Sex (0 = Female, 1 = Male) 


0.87 


0.33 


0.88 


0.33 


ASVAB Scores 










ASVAB Administrative 


59.62 


18.47 


59.93 


18.47 


ASVAB Mechanical 


. 58.97 


25.88 


57.19 


22.90 


CRITERION VARIABLE 










First Attempt Lesson Time 


84.90 


64.74 


80.10 


62.36 
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variance accounted for amounted to 5,34 percent. The results of the cross valida- 
tion were considered to be quite satisfactory. It may be concluded that, when 
applied to data comparable to that used for equation generation, the regression 
equation model can satisfactorily predict first-attempt-lesson-times. 

The question remained, however, as to the extent to which the model would 
provide satisfactory predictions of first-attempt-lesson-times for a new course, 
one containing a different type of subject matter.' That is, to what extent could 
the model be generalized to courses other than those employed in the data base? A 
thorough evaluation of the model's generality would have required the construction 
of a new data base and was, unfortunately, beyond the scope of the current study. 
However, a partial test could be made through use of the available data base. 

It will be recalled that the data base sampled a total of sixty-four lessons 
from two courses: twenty-nine from IM, and thirty-five from WM. The question of 
model generality was rephrased to ask to what extent did unmeasured characteristics 
of these two courses, i.e., factors specific to each course, contribute to the 
variability in first-attempt-lesson-time. That is, to what extent could the 
variance accounted for be increased by knowledge of the course from which the 
lessons and students were drawn? The degree to which knowledge of course member- 
ship increased predictability could be taken as a rough measure of the limitation 
of the model generality. 

A linear models comparison was employed in which the restricted model, Model 10, 
was defined as containing the 14 variables in the final regression equation. 
A full model. Model 11, was defined which contained, in addition, a variable 

2 

representing course number (IM = 1, WM = 4). The full model resulted in an R of 

2 

.41300. The restricted model, the final regression equation, produced an R value 
of .3948, a reduction of .01820. This represented a significant reduction 
(£ (1 , 2958) = 91.71 , .001). Thus, specific course membership accounted for an 
additional 2.0 percent of the variance. 
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Although significant, the amount of variance accounted for by knowledge of 
course membership is relatively small. Although some loss in predictability could 
be expected if the regression equation model were applied to a new course, the 
results of this final comparison imply that the accuracy of prediction would not be 
seriously diminished. 

# 

4.3.6 Generation of Block-Completion-Times from First-Attempt-Lesson-Times - As was 
discussed in paragraph 4.2.1, Definition of the Criterion Variable^ the criterion 
variable of interest was block-completion-time. It was, however, necessary to 
develop the regression equation at the lesson level rather than at the block level 
in order to obtain sufficient variability in content. Having generated the 
regression equation, it was now necessary to provide a transition to block level 
prediction. As was outlined in paragraph 4.2.1, the approach employed was to 
determine a proportional "lost time" for block completion and the sum of the 
completion times for all the lessons contained in the block. 

As was discussed in paragraph 4.2.3 of this report, none of the nine block 
samples from the tv;o courses contained a complete set of lessons. This was due to 
the fact that there were AIS lessons in each block which represented new content, 
content for which no conventional instruction POIs were available. Since there 
was no basis for establishing course content predictor variables for these lessons, 
they were excluded from consideration. Since the data base constructed for this 
study did not contain any examples of complete blocks, it was necessary to return 
to the AIS database to obtain the required information, i.e., block-completion- 
times and first-attemot-times for aJl of the lessons in each block. 

Data were retrieved from each of the nine blocks employed in the current study. 
A student's data were included in this analysis only if reliable first-attempt- 
lesson-times were available for all lessons in the block, both those included in 
the original analyses and those which had previously been excluded. Furthermore, 
a student's data were excluded if (a) the recorded block-completion-time was less 
than the sum of the first-attempt-lesson-times or (b) if block-completion-time 
exceeded reasonable maximums identified by the AIS materials developers. For each 
student who met these constraints, two values were generated: (a) the sum of 
first-attempt-lesson times; and (b) block-completion-time. The difference between 
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these two values was then expressed as a proportion of the sum of first-attempt- 
lesson-times. This constituted the "lost time". ratio for that student. A mean lost 
time was then calculated for each of these nine blocks. A final mean was determined 
on the baiis of these nine values. The reason for computing the mean in two steps 
rather than simply summing across blocks was to avoid biases introduced by unequal 
numbers of observations in the various blocks. 

The resultant lost time r^tio was 0.40. That is, on the average, block 
completion time was found to equal 1.40 x (sum of first-attempt-lesson-times). 
As may be seen by reference^ to the Descriptive Model (Figure 2), this unmeasured 
lost time was comprised of the time attributable to the following activities: 

(a) queuing for lesson resources and test scoring (relatively negligible times); 

(b) remedial sti'dy, queuing and retesting on failed lessons; (c) administrative 
overhead; (d) credentiallihg testing and queuing for test scoring; and (e) block 
level remediation following failure of a credentialling test. In the Cost Model, 
to be discussed in Section 5.0 of this report, the product of the regression 
equation was multiplied by 1.40 to obtain an estimate of block-completion-time. 
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5.0 THE COST MODEL 

¥ — • ' 

One of the ubjectiveb of this study was the modification of a cost analysis 
computer program previously developer* by the Rand Corporation. Speoif ica.1 ly, the 
Rand model was reprogrammed to use t egression equation developed in this study 
as the method for generating predict:., o of course completion times for students. 
The unmodified Rana model uses a completion time.enter'ed by the user without speci- 
fying how this time was estirftated. 

The Rand MODIA cost model (Hess and Kanter, 1976) i? ojiually comprised of 
ttjree computer programs: a user interface program (UI); a resourb.^ utilization 
model (RUM); and a cost analysis program (MODCOM). The UI is "an interactive pro- 
gram which produces a course description in computer-compatible data which can be 
input Into the RUM. The RUM is a batch program which produces detailed course 
operati n reports regarding student flow patterns and waiting times as well as 
resource demand and utilization. The UI and RUM are designed to be used repeatedly 
until the course planner has generated several course designs. The costs of these 
alternative course descriptions are then compared by the bdtch process cost model 
program (MOncOM). Resource requirements from the UI, RUM, user sypplied course 
completion times; and cojt and manning data are input into MODCOM to produce total 
cdurse costs for up to five years. On some of the cost and rr.anning factors, default 
v-»lues are used if not supplied by. the analyst. 

The computer program develope'i in this study is named Aid for Instructional 
development and Evaluation (AIDE).' It i?^ modification of the MODCOM cost 
analysis program and retains all of the cost analysis, features and outputs of 
MODCOM while allowing for course completion time to be estimated by the regression 
equation developed in this study. 

Although AIDE inputs are sufficiently documented in this report for it to 
be used by itself, it is designed to be used as a replacement for MODCOM in the 
MODIA series of programs, and the detailed MODCOM documentation should be used with 
AIDE. 
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5 . 1 DES^.^ IP TION AND USE OF THE AIDE PROGRAM 

The AIDE program projects the investrrent and operating costs of a 
given course design for a five-year time frame. Course completion time 
for each u of student is computed and used to calculate student pay 
and allowances costs. The number of student entrants per year and their 
completion times are used to determine resource requirements and costs.. 
Resources include manpower, courseware, hardware, and facilities. These 
resources require specification in term: of quantities such as attrition rates, 
and utilization rates. These specifications are outputs of the MODIA.UI and 
RUM programs or may be supplied from knowledge of a specific course. Flexibility 
in the inputs required allows the model to be used at any level of detail 
desired. 

The following outputs are generated by the AIDE program: 

0 Input Data Listing 

0 Input Lesson Descriptor Values Listing 

0 Course Completion Data Generated by the Regression Model 

0 Personnel Distributions and Cost Factors 

0 Graduate Summary 

0 Manpower Summary 

0 Courseware, Hardware, and Facilities Requirements by Year 
0 Functional Cost Summary. 

These outputs are illustrated in Figures A-1 throup*^ a -8 in Appendix 
A. AIDE outputs 4 through 8 are identical with the or^'qinal MODCOM out- 
puts. Output 1 differs only in that the additional . :lt::it and lesson 
variables unique to AIDE' are listed. ^ Outputs 2 and 3 are unique to AIDE. 
Output 2 provides a complete list of the irput lesson descriptor variables, 
^-^nd Output 3 is a summary of the course completion data generated by the 
regression equation. 

A descrl ' ; of input data preparation is given in Appendix A. A 

more detailed .scription of the data input is provided by the RAND MODCOM 

documentation. Any differences in input data preparation between AIDE 

and MODCOM are noted specifically in Appendix A. In general, the differences 
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between the two prografiis are that the A-IDE regression equation requires 
input of values for three student descriptor variables for each type of 
student defined and values for the eleven course descriptor variables. 
The particular regression equation developed in this study is currently 
hardcoded in the AIDE program, but AIDE could be changed to employ any 
similar regression equation with only minor changes to the FORTRAN code 
(as noted in Appendix A). 

One limitation of the AIDE cost analysis program is that, while there 
is variability in the predicted course completion times generated by the 
regression equation, no corresponding variability is computed for course 
costs by the cost model . 

Although AIDE can take up to 75 different lessons as defined by their 
lesson descriptor variables, it should be noted that the original (con- 
ventional course) lesson lengths in this study were app>^ox imately one to 
seven hours. Thus, the prediction equation for individual izedo lesson and 
completion times should not be generalized to lessons whose original 
lengths fall outside this range. Regarding input data preparation, 
original course lengths outside of this range should be broken down into 
sums of lessons whose lengths fall within the above ranged- 

5.2 EX^^- MPLE USE OF AIDE 

To illustrate how the AIDE program can be exercised, course completion 
and cost data were generated for a standard sample and for additional samples 
representing three specific questions discussed in paragraphs 5.2.1 through 
5.2.3. 

Input data for all of these samples were obtained primarily from example 
values given in the MODIA documentation, but u number of values were repre- 
sentative of available AIS da.ta. Specifically, these values were (a) a 
conventional course length of 50 hours, (b) a student washout rate of 10 
percent, (c) a student entry interval of once per 30 hours, (d) a total 
student entry rate of 3200 per year or approximately 60 per week, (e) 
lesson descriptor variables, and (f) student charav teristic variables. 
For the standard sample, an approximate normal distribution of §tudent 
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variable values was chosen for the ASVAB Administrative (ASVADM) and 
Mechanical (ASVAMEC) scores. The mean values of these two distributions 
were set equal to the mean values of Sample One from the regression 
analysis section of this study. Sex was held constant by defining an all 
male sample. Likewise, student personnel type, designation, and paygrade 
value were held constant for all students at values derived from the 
MODIA documentation. 

Figures A-1 through A-8 in Appendix A illustrate the AIDE output for 
this standard sample. Figure A-2 lists the values of the lesson variables, 
and Figure A-3 lists the values of the student variables. The standard 
sample tims and cost results are summarized in Table 7. The first portion- 
of the table, Course Completion Data, is derived from the AIDE program 
Oi'tput 3 (Figure A-3 w Appendix A). It lists the expected course 
completion times for students of five differing ability levels as measured 
by the ASVAB Administrative (ASVADM) and Mechanical (ASVMEC) scales. The 
avv^rage expected course completion time across all students is shown to 
be 19.17 hours, a savings of 52.1 percent from the conventional course 
length of 40 hours. The second portion of the table, the Functional Five 
-Year Cost Summary, is derived from AIDE Output 8 (Figure A-8 in Appendix 
A). The figures shown for each category are totals over the ini.tial five' 
year period assuming the distribution of studerit ability levels shown in 
the top portion of the" table. The various cost items are described on 
pages A-27 and -28 of Appendix A. 
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COURSE COMPLETION DATA jlH HOURS) 



Student Variables 
ASVADM ASVMEC 



84 
72 
60 
48 
36 



85 
72 
59 
46 
33 



Course 
Completion 
Time (Hrs.) 



17.0 

18.1 

19.2 

20.2, 

21.3 



Average Course Completion Time is 19.17 Hours 
Average time Savings is 52.1 Percent 
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FUNCTIONAL FIVE YEAR COST SUMMARY 

Courseware Procurement 

Printed Media $136,700 
Display Media 48,000 
Software 13,100 

Hardware Procurement 

Media -Hardware 22,400 
Special Equipment , 0 

Overhead Hardware 2,000 

Facility Construction 75,000 

Pay and Allowances 

Students 10^1,700 

Instructors 1644,800 

Curriculum Personnel 79,300 

Facilities Maintenance Personnel 204,500 

Training Admin. Personnel 60,000 
Base Operating Support Personnel ' 335,000 

Medical Personnel 59,100 

PCS Costs 

Students 7165,700 
Instructors 25,400 

Instructor Training 

Factory Training of Initial Cadre 3,000 
Education Training 54,200 

Miscellaneous Operating Costs 

Computer Service Charges 
. Hardware Replenishment/Repair 
Miscellaneous Supplies 



TOTAL COURSE COST 
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•It should be undf rstood that the purpose here was to illustrate how 
the cost model can be exercised, not to provide an accurate description 
of the cost structure of an AIS course. If the data were to be representative 
of an AIS course, it would be necessary to increase the number of lessons 
in the sample from ten to about 120, and a number of additional inputs would 
need to be modified to be representative of either the IM or WM course 
characteristics and students. 

5.2.1 Effect of Variation in Student Aptitude Level ~ The first question 
investigated asked how- cost would be affected if the average level of 
student aptitude is increased or decreased where aptitude is measured by 
the ASVAB Administrative and Mechanical scores. The purpose of this 
particular question was to represent a situation in which an admln'^ ^rctc^ 
might wish to learn the effects on course costs of lowering course entr^ 
^requirements possible outccnie of zero draft) or' of raising them (a 
possible outcome of increased unemployment in the civilian sector). 

Tv/o more distributions of student variables were generated which 
were also approximately normal and whose ASVADM and ASVMEC means were 
two-thirds .of a standard deviation below and one-half of a standard 
deviation above the mean of the standard sample, respectively. The dis- • 
tributions employed (including the standard sample) are illustrated in 
Table 8. For each sample, the same number of students v/as input for each 
of five years, and all other input variables were held constant. With an 
entry interval of one week (30 hours), the entry rate was 64 student.- p.er 
week, roughly the number of weekly student entries in the AIS IM course at 
Lowry AFB. 
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TABLE 8 

DISTRIBUTIONS OF STUDENT VARIABLES FOR COMPARISON OF 
STANDARD. BELOW AVERAGE, AND ABOVF AVERAGE STUDENT POPULATIONS 



SAMPLE DESCRIPTION 


ASVADM 


ASVMEC 


QTMnCMT CMTDAMTQ DFD VFAD 
olUUtlil t(ilKM(ilO rtK T LMK 




QA 
OH 




?nn 




"70 

72 


1 L 


pfin 


Standard Sample 


60 


59 


1200 




48 


•46 


800 




36 


JO 






yo 


QP 


200 




84 


85 


pnn 

oUU 


Above Average Sample 


72 


72 


1200 




60 


59 


800 




48 


46 


200 




72 


72 


200 




60 


59 


8O9 


Below Average Sample 


48 


46 


1200 




36 


33 


800 




24 


20 


200 



The time and cost results for the two Above and Below Average student 
samples are given in Tables 9-1 and 9-2, respectively. These Tables are in 
the same format as Table 7, and the data were derived from the same AIDE 
program outputs, i.e., Output 3 and Output 8. 

For purposes of comparison, data from Tables 7, 9-1 and 9-2 are summarized 
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STUDENT VARIABLES 
ASVADM ASVMEC 



96 
84 
72 
60 
48 



85 
72 
59 
46 
33 



COURSE 
COMPLETION 
TTMF (HRS.^ 

16.7 
17.8 
18.9 
20.0 
21.0 



Average Course Completion Time is 18.88 Hours 
•Average Time Savings is 52.8 Percent 



FUNCTIONAL FIVE YEAR COST SUMMARY 

Courseware Procurement 

Printed Media 
Display Media 
Software 



$ 136,700 
48,000 
13,100 
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Hardware Procurement 

Media Hardware 
Special Equipment 
Overhead Hardware 

Facility Construction 

Pay and Allov/ances 

Students 
Instructors 
Curriculum Personnel 
Hardware Maintenance Personnel 
Facilities Maintenance Personnel 
Training Admin. Personnel 
Base Operating Support Personnel 
Medical Personnel 

PCS Costs 

Students 
Instructors 

Instructor Training 

Factory Training of Initial Cadre 
Education Training 

Miscellaneous Operating Costs 

Computer Service Charges 
Hardware Reple> ishment/Repair 
Miscellaneous supplies 

TOTAL COURSE COST 
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20,900 
0 

1,800 
75,000 



1075,700 
1644,800 
79,300 
195,000 
60,000 
335,000 
273,300 
59,100 



7166,700 
25,400 



3,000 
54,200 



330,000 
7,500 
4 2,900 

$11,647,400 
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AIDE TEST RUN FOR BELOW AVERAGE STUDENT SAMPLE 

COURSE COMPLETION DAT A (IN HOURS) 

STUDENT VARIABLES COURSE 

COMPLETION 

ASVADM . ASVMEC TIME (HRS.) 
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72 
60 
48 
36 
24 



85 
72 
59 
46 
33 



17.3 
18.4 
19.5 
20.5 
21.6 



Average Course Completion Time is 19.45 Hours 
Average Time Saving is 51.4 Percent 



FUNCTIONAL FIVE YEAR" COST SUMMARY 
Courseware Procurement 



Printed Media 
Display Media 
Software 

Hardware Procurement 
Media Hardware 
Special Equipment 
Overhead Hardware 

Facility Construction 

Pay and Allowances 

Students 

Instructors 

Curriculum Personnel 

Hardware Maintenance Personnel 

Facilities Maintenance Personnel 

Training Admin. Personnel 

Base Operating Support Personnel 

Medical Personnel 

PCS Costs 

Students 
Instructors 

Instructor Training 

Factory Training of Initial Cadre 
Education Training 

Miscellaneous Operating Costs 

Computer Service Charges 
Hardware Replenishment/Repair 
Mi<;cellaneous Supplies 

TOTAL COURSE COST 



$ 136,700 
48,000 
13,100 

22,400 
0 

2,000 
' 75,000 



1108,100 
1644,800 
79,300 
204,500 
60,000 
335,000 
278,300 
59,100 



7166,700 
25,400 



3,000 
54,200 



330,000 
8,000 
43.900 

$11,697,500 
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in Table 10. may be seen from Table 10, dramatic variations in aptitude 
in the student population, as measured by the ASVAB Administrative and 
Mechanical scores, have relatively little effect on course completion time, 
approximately 1.5 percent. This is due to the fact that, in the 
regression equation employed (see Table 5), these two variables, although 
significant, account for relatively little of the variability in first- 
attempt-lesson-time. The effect on Total Course Cost is even less, 
approximately 0.2 percent. This is due to the relatively large fraction 
of the course costs which are considered by the model to be fixed rather 
than dependent on variable course length. 



TABLE 10 

TIME AND COST COMPARISONS FOR DIFFERENT STUDENT DISTRIBUTIONS 



SAMPLE DESCRIPTION 


AVERAGE TIME 


STUDENT PAY COST 


TOTAL COI 


JRSE COST 


Above Average 
Standard 
Below Average 


Hours 


% Change 


$* 


% Change 


$* 


% Change 


18.88 
19.17 
19.45 


-1.5 
0.0 
1.4 


1075.7 
1091.7 
1108.1 


-1.5 
0.0 
1.5 


11647.4 
11675.6 
11697.5 


-0.24 
0.00 
0.19 



* In Thousands of Dollars 



5.2.2 E ffect of Variation of Proportion of Memory Versus Cognitive Content - 
The second question investigated the effect of variations in the relative 
proportions of Memory to Cognitive content. The purpose of this question 
w?'^ to illustrate the differing outcomes that may be expected when courses 
with different characteristics are individualized. The variables in the 
regression equation (Table 5) indicated that individualization of a course 
containing a relatively large amount of memory content would result in 
more substantive time savings than would individualization of an other- 
wise similar course with less memory and more Cognitive content. 
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To investigate this question, the n'jn^ber of minutes of. memory content 
material (MEM) in the course represented in the standard sample was 
increased by one standard deviation, i.e., 72 minutes per lesson. In 
order to keep lesson lengths constant, a corresponding reduction of 72 
minutes per lesson was made in the amount of cognitive content (COG). This 
represented a reduction of 0.49 standard deviations in the cognitive content. 

The. Time and Cost results for this high-memory-content course 
are given in Table 11 which employs the same format as Table 7. For 
purposes of comparison with the standard sample, data from Tables 7 and 11 
are summarized in Table 12. As may be seen from Table 12, substantially 
greater time savings would be expected from individualization of a course 
with a higher percentage of memory content. While the predicted average 
course completion time for the standard sample course was 19.17 hours 
(down from 40 hours as taught by conventional methods), the predicted 
average course completion time for the high-memory-content course (also 
assuming a 40 hour conventional course baseline) was only 13.51 hours. 

Again, it should be noted that while there are large differences 
in student .pay costs between the two types of courses (29.5 percent), the 
difference in Total Course Cost is relatively small (8.1 percent). 
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TABLE 11 

AIDE TEST RUN FOR HIGH-MEMORY/LOW-COGNITIVE-CONTENT COURSE 
COURSE COMPLETION DATA (IN HOURS) 
STUDENT VARIABLES COURSE 



COMPLETION 

' ASVADM ASVMEC TIME (HRS.) 



84 85 11.4 

72 72 12.4 

60 59 13.5 

48 46 14.6 

36 33 15.7 



Average Course Completion Time Is 13.51 Hours 
Average Time Saving Is 66.2 Percent 

FUNCTIONAL FIVE YEAR COST SUMMARY 
Courseware Procurement 



Printed Media $ 136,600 

Display Media . 48,000 

Software 13,100 

Hardware Procurement 

Media Hardware 13,400 

Special Equipment . 0 

Overhead Hardware - 300 

Facility Construction 75,000 

Pay and Allowances 

Students '769,300 

Instructors 1252,800 

Curriculum Personnel 79,300 

Hardware Maintenance Personnel 142,500 

Facilities Maintenance Personnel 60,000 

Training Admin. Personnel 289,500 

Base Operating Support Personnel 205,300 

Medical Personnpl ' ^3,100 

PCS Costs 

Students 7166,700 

Instructors 19,200 

Instructor Training 

Factory Training of Initial Cadre 3,000 

Education Training 40,700 

Miscellaneous Operating Costs 

Computer Service Charge 330,000 

Hardware Replenishment/Repair 5,500 

Miscellaneous Supplies 32,400 

TOTAL COURSE COJT $10,725,100 
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TABLE 12 

TIME AND COST COMPARISONS FOR STANDARD AND HIGH MEMORY/LOW COGNITIVE 

CONTENT COURSES 



SAMPLE DESCRIPTION * 


AVERAGE TIME 


STUDENT PAY COST 


TOTAL COURSE COST 




Hours 


% Change 


$* 


% Change 


$* 


% Change 


Standard Sample with 
Mean Values on MEM, 
COG 


19.17 


0.0 


1091.7 


O.U 


11675.6 


0.0 


Standard Sample with 














MEM increased 1 S.D. , 


13.51 


-29.5 


769.3 


-29.5 


10725.1 


"-8.1 


COG decreased 0.49 S.O. 















* In Thousands of Dollars 



5.2.3 Effect of Variation of Conventional Course Presentation Methods - One 
of the most striking aspects of the regression equation to predict first- 
attempt-lesson-time (Table 5) was the difference between the coefficients 
of two of the quantitative course content variables: Lecture/Discussion 
Minutes and Demonstration/Performance Minutes - Cognitive . According to 
these coefficients, each additional minute of Lecture/Discussion 
presentation in the conventional course would be expected to increase 
first attempt lesson time in the individualized course by only 0.12 
minutes. In contrast, each additional minute of Demonstration/ 
Performance-Cogni tjve presentation would be expected to increase first- 
attempt-lesson-time in the individualized course by 0.26 minutes. On 
the assumption that these values were approxima :ely cor^^ect, a final 
comparison was made to illustrate the differing effects of individual- 
izing two courses which differed on these dimensions. 

For the purpose of this comparison, the number of minutes of 
Cognitive Demonstration Performance time (CDPMC) was increased by 0.75 
standard deviations (48 minutes) per lesson. A corresponding reductiori 
of 48 minutes per lesson in Lecture/Demonstration time (CLDM) amounted 
to a change of 0.49 standard deviations. 

.5-13 31 

ERIC 



EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUME II 



MDC El 570 
15 MAY 1977 



The time and cost results for tItJs high-demonstration/performance- 



method course are given in Table 13 which employs the same format as Table 
7. For purposes of comparison with the standard sample, data from Tables 7 
and 13 are summarized in Table 14. As may be seen from Table 14, no time 
savings would be expected from the individualization of a course which 
employed Demonstration/Performance methods for this very high proportion 
of the content. The predicted average course completion time for the 
Individualized high-demonstration/performance-method course was 44.52 hours, 
an increase of 4.52 hours over the conventional course. 
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TABLE 13 

AIDE TEST .RUN FOR HIGH-DEMONSTRATION/PERFORMANCE-METHOD COU'^SE 

COURSE COMPLETION DATA (IN. HOURS) 

• • SiUDENT VARIABLES ^ COURSE 

COMPLETION , • 
ASVADM ASVHEC TIME (MRS.) 



ERIC 



84 
72 
60 
48 
35 



85 
72 
C9 
46 
33 



42.^ 
43.4 
44.5 
45.6 
46.7 



Average Course Completion Time Is 44.52 Hours 
Average Time Sa,ving i-s' -11.3 Percent 

FUNCTIONAL FIVE YEAR COST SUMMARY 



Coursev;are Procurement 

Printed Media 
Display Media 
Softwa re 

Hardware Procurement 

Media Hardware ' • 

Special Equipment 
■ Overhead Hardware 

Facility Construction 

Pay and Allowances 

Students 
Instructors 
Curriculuir Personnel 
Hardware- Mai ntenence Person.iel 
Facilities Maintenance Personnel 
Training' Admin. Personnel 
Base Operating Support Personnel 
Medical Personnel 

PCS Costs 

Students 
In'stnictors 

Instructor Training 

Factory Tnining of Initial Cadre 
Education Training 

Miscellaneous Operating Costs 

Computer Service Charijes 

Ha" rd wa re Re p 1 e ri i £ hme n t/ Rep a.]' r 

Miscellaneous Supplies 



TOTAL COURSE COST 



$ 137,80li 
48,000 
13,100 



51 ,400 
0 

12,800 
150,000 



2531,600 
3751,700 
79,300 
463,600 
115.500 
544,300 
607,400 
131 ,500 



7154,300 
67 400 



3,000 
125,400 



410,000 
17,500 
95,600 

$16,511 ,200 
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TABLE 14 

• TIME AND COST COMPARISONS FOR STANDARD AND HIGH-DEMONSTRATION/ 

■ PERFORMANCE-METHOD COURSES 



SAMPLE DESCRIPTION 


AVERAGE TIME 


STUDENl PAY COST 


TOTAL CO 


URSE COST 


Standard Sample v/ith 
Mean Values on 
CDPMC, CLDM 


Hours 


% Change 


$* 


% Change 


$* 


% Change 


19.17 


0.0- 


1091.7 


0.0 


11675.6 


0.0 


Standard Sample with 
CDPMd increased .75 
S.D. CLDM decreased 
.49 S.D.- 


44.52 


1?2.2 


2531 .6 


131.9 

1 


16511.2 


41 .4 



* In Thousands of Dollars 



, 5.3 DISCUSSION OF AIDE EXAMPLE RUN RESULTS 

The' three questions discussed above represent simple examples of how 
the AIDE cost^ model analysis program may be exercised, using differing 
, student atid lesson variable values, to investigate the expected cost 
savings resulting from individualizing different types of courses. Again, 
it should be emphasized that the data presented were not intended to be 
representative of the AIS courses from wh'.ch the regression equation data 
base was derived. Neither was the intent to exploit fully the capabilities 
of the Rand MODCOM' program from which the AIDE program was derived. 

There are a number of general points which should be explicated. 
The various example runs indicate that direct student costs, pay and allow- 
ances, account for relatively little of the'total course costs. For the 
Standard Sample (Table 7), student pay and allowances account ,or only 
9.35 percent of the total. Although, as is discussed below, some of the 

.cost figures provided are somewhat misleading, these values do illustrate 
the point that military teciinical training does hav •-■ bstantiai '^'xed 
costs which are relatively insensitive to reductions in training time 
per se . 
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These additional costs were, however, inflated by two pre-analysis 
decisions reflected in the input data. First, the input data indicated 
tha^ student relocation for training was to be treated as a permanent change 
Qf station (PCS). Given the predicted average course length of 19.17 hours 
(Standard Sample, Table 7), it would be more cost-effective to handle 
student relocation throjgh temporary assignment - TOY. PCS costs for the 
standard sample were $7,166,700 over the five, year period; six and one-half 
times as much as student pay and allowances over the same period. If 
rrlocatibn for training had been handled by means of TOY assignment and 
if one assumed a six hour training day, this expense would have been reduced 
to approximately $2,9p3,5Q0. This figure still represents over two and ^ 
one-half times the cost of student pay and allowances. 

A second factor which increased predicted Costs was the decision that 
the entry interval should be 30 class hours or once per week* Given that 
the average course completion time was predicted to be less thr^n 20 hours, 
very few students would still be present in the course at the end of the 
week. While the average student load was computed to De 36.2, the actual 
number would range from a high of at least d4 students at the time of a 
class entry to a low of near zero just prior to the next class entry. This 
represents an extremely inefficient use of both personnel and facilities. 
Although such a test run was not made, it can be anticipated that a 
substantive cost reduction would result ir the entry interval was reduced 
to six hours, allowing 12 to 13 new students to enter the course each day. 

Although these two factors, permanent change of station and once per 
week entry, did inflate the estimated costs relative to student pay and 
allowances, the conclusions drawn concerning the relative cos^ts of the' 
various samples run, i.e., above and below average student ability levels, 
high-memory/low-cognitive content, etc., remain valid. Even with TOY 
assignment and daily course entry, student pay and allowances would still 
contribute only a fairly small proportion to the total course costs. 
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6.0 CONCLUSIONS AND RECOMMENDATIONS 

This study represents the first ef-^ort to employ data from an operational 
environment in an attempt to relate variables such as course content, instructional 
and system design considerations, and student character;istics to completion time in 
a self-paced course and to the cost of developing, implementing, and maintaining 
such a course. The study has, we believe, demonstrated that a viable time savings 
and cost model can be developed given a sufficient data base from which to draw 
information. The generality and precision of such a model is, however, dependent 
on the depth to which the instructional system from which the data are derived 
can be fully and explicitly described. Analogously, the utility of the model for 
a given application will depend on the accuracy with which the course being con- 
sidered for self'-pacing is described. In addition to a description of the course 
content, the model requires a number ^f assumptions concerning instructional and 
system design variables and the characteristics of the students to be enrolled in 
the course. The predictive accuracy of the model will, of course, always be limited 
by the validity of these assumptions. 

As was stated above, the authors believe that the current study has demon- 
strated the feasibility of developing time savings and cost models which have 
practical utility. It must be recognized, however, that there is a number of 
limitations both to the specific model developed here and to future efforts which 
employ this work as a starting point. Each of the three major components of the 
overcll effort, tne descriptive model, the predictive model, and the cost model, 
will be discussed with respect to these limitations as well as with respect to 
their positive aspects. 

The descriptive model of computer-based training systems can be viewed as 
serving as a framework for the predictive model and a guide to its development. 
The utility of the oredictive model will always be limited by the generality and 
completeness of the descriptive model. While the descriptive model emplo>od in 
this study would appear to be appropriate, it must be recognized that its adequacy 
was not actually tested by the current effort, since the predictive model developed 
directly addressed only a small portion of the total descriptive model. 
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It is suggested, however, that the general characteristics of the descriptive 
model proposed are appropriate to 'an effort of this type, i.e., description of an 
instructional syste.^ in terms of a number of time-consuming components and the 
probabilities associated with a limited number of decision points. Our experience 
with predictinga single time component (first-attempt-lesson-time) indicates that 
it is feasible to develop miniature predictive models for the remaining components, 
given the availability of reasonably accurate data. 

The one exception to this generally optimistic view must be noted - prediction 
of the time component entitled Cumulative Administrative Overhead, , "''he quantity of 
administrative overhead would appear .to be heavily dependent on locai course 
policies. While this assumption needs to be examined i» greater detail, it would 
appear that the time attributable to this component may be too specific to local 
situations to be estimated with any degree of accuracy on t.he basis of available 
data. 

A second problem area lies in estimating the probabilities associated with the 
pass/fail decisions following the various lesson and unit tests. As was discussed 
in Taragraph 4.2.1 of this report, these decisions again appear to be Highly dependent 
on local course policies. Consequently, predictions based on data from any given 
course or set of courses will be biased by the policies in effect in those courses 
and fail to generalize to a new course, "^l^e best solution to this problem may be to 
establ'^sh tolerable failure rates as part of the instructional syst?m design speci- 
fications and assume that instructional materials and procedures will be developed 
such that student performance will meet these specifications. 

It should be noted that the descriptive model can be viewed as consisting of 
several nested loops: the time required to complete the major instructional unit; 
the time required to complete a given lesson; and the time required by the indi- 
vidual components within the lesson completion loop. As a result, the model has the 
interesting attribute that it is not necessary that all ut the individual components 
be modeled explicitly. To the extent that individual components are modeled 
accurately, the overall estimate of instructional unit completion time becomes more 
responsive to vaMa 'ons in content, student characteristics, and instructional 

method. The model still has utility, however, if only gross .estimate? are ^nade of 

i 

the total time required by the set of components which were not modeled. The 
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initial effort represented by this studv exemplifies the extreme case of such an 
approach. The single component, first-attempt-lesson-time, was modeled as accu- 
rately as possible under the circumstances. The cumulative time attributable to 
the remai'ning components was then estimated as a function of first-attempt-lesson- 
time. The final result, although not as accurate as might be des-'Ved, does provide 
a useful estimate of the total time required to complete an arbitrary instructional 
unit. Detailed model ng of any additional component will increase the accuracy of 
the overall estimate. Thus, further refinement of the model can proceed step-by- 
step as more data become available. \ 

Discussion of the predictive model brings us to tne actual mechanics involved 
in estimating the time requirad for an individual studei.t to complete an arbitrary 
instructional unit. It is of some interest to compare the methodology and results 
of the current study with the work reported by Wagner, Behringer and Pattie (1973) 
in predicting individualized course completion. The major differences betwe-^n the 
two efforts are a function of their differing objectives. While Wagner, et aj , 
were interested in p^^edicting the time required to complete a particular self-paced 
course, the intent of the work reported here was to predict ttie time required to 
complete an arbitrary instructional unit, given a limited set of characteristics of 
that unit. Specifically, whereas all of the students in the Wagner, et al, study 
worked through the same set of materials, the criterion variable in the current 
study represented the time to complete each of 66 different instructional units. 
The fact that the multiple correlation coefficient of .6283 obtained here compares 
favorably with the correlations of .65 to .75 reported by Wagner, et al , indicates 
that the lesson descriptor variables employed in the current study did accurately 
represent the characteristics of the various lessons. While Wa^jner, et al , were 
able to obtain substantially higher correlations (.85 to .87 through the use of 
within-course measures, such measures would not have been appropriate to the objec- 
tives of the current study. Wagner, et al , also found that the measurement of 
individual difference characteristics through the use of course content-related 
instrume*nts was preferable to the us^e of the standard military battery. The 
current study found that the use of available preassessment measures did not appre- 
ciably increase the accuracy of prediction in the presence of ASVAB scores, but the 
results Pight have been different had the preassessment battery contained more 
content-specific tasks. Again, however, such content-specific measures would have 
very limited utility given the objectives of the current work. 
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The regression analysis approach employed in this study would also appear to 
be appropriate for estimating the time required for many of the remaining components 
of ti'^e Descriptive Model such as Remedial Study Time on Lesson i and the time 
required for the various diagnostic and credentual 1 ing tests. In other cases, such 
as estimating the time consumed in obtaining resources and test scoring, an alter- 
nate approach based on queuing theory might well be more appropriate. 

It should be noted that the current study did not fully investigate all aspects 
of the regression model approach. Only simple, linear predictors were employed. 
There is reason to believe that accuracy of prediction could have been imp^roved if 
the effects of nonlinear transformations of the predictor variables had been inves- 
tigated. For example, it is reasonable to expect that the relationship between 
first-attempt-lesson-time and student individual differences could be better repre- 
sented by a curvilinear relaticnship than by a simple linear relationship. The 
failure to investigate interactions among the various predictor variables represents 
an even more serious limitation. The final regression model does not include any 
terms which are products of two or more predictor variables. As a result, for exam- 
ple, the final regression model contains the assumption chat while two students of 
differing ability levels will complete a given lesson in differing amounts cf time, 
the difference between their two completion times will be independent of the amount 
of lesson content. That is, if Student A requires ten more minutes than does 
Student B to complete a particular lesson, the model implies that Student A will 
also require only ten mo^e minutes than Student B to complete a second lesson con- 
taining twice ai, much content as the first lesson. The incongruity would be resolved 
if the predictor set contained one or more variables which were the multiplicative 
products of appropriate student ability and course content measures. The same line 
of argument implies the desirability of including product terms which incorporate 
instructional design variables such as the number of self-check items. 

The effort required for adequate investigation of potential transformations and 
product terms was simply not feasible within the temporal and fiscal constraints of 
the current study. Investigation of at lea^^ interaction terms should certainly be 
given consideration in any work directed at refining the predictive model. 

The major limitations of the current studv can be traced directly to the lack of 
a sufficiently broad and reliable data base. To a great extent, any future efforts 
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in this area will be limited by these same constraints. In general, data acquisition 
problems fall into four broad categories: self-paced student performance data; base- 
line course content descriptions; the variability of examples of instructional 
methods; and the distribution of observations across variations in course content, 
instructional methods, and student individual difference characteristics. 

At the time the data collection effort for the current study was being con- 
ducted, the AIS was in a relatively rudimentary stage of development, and the only 
reliable lesson level time data available were first-attempt-lesson-times. There- 
fore, the predictive model was limited in explicit prediction to this single 
component. Since that time, further development of the AIS has allowed collection 
of reliable, remedial study times and lesson completion times. Were the study to 
be repeated now, a predictive model could be developed which addre:)Sed several com- 
ponents of the instructional process. Obviously, future efforts will be more 
fruitful if they are conducted in the context of an instructional system with a 
fully developed data collection and recording conoonent. 

While there are advantages to conducting future muueling efforts in the context 
of an established system, there is a definite disadvantage in attempting to model 
courses which have been conducted under the auspices of such a systen for any length 
of time. The modeling approach employed in this study assumes the existence of 
course planning documents which pertain to the course as it was taught in a conven- 
tional, group-paced mode. Since technical training courses do change, there is, 
with time, less and less overlap between the current content of the self-paced course 
and its content when it was taught in the conventional mode. It will be recalled 
that none of the nine blocks of instruction sampled in this study contained exactly 
the same content as they did in the earlier, conventional course, ^f the study were 
to be repeated now, there would be even less overlap. Proposed changes to the 
Inventory Management course indicate that within a year almost none ot the content 
in the original, group-paced course will still be taught. It is therefore suggested 
that any future efforts at model refinement be conducted in the context of an estab- 
lisher system to which new courses are being adcied . 

There is undoubtedly a tendency for a giver, instructional system to shape the 
instructional methods employed in that system. One tends to find relatively little 
variability in method across lessons. Thus, the effect of advanced organizers such 

6-5 90 




EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUME II 



MDC El 570 
15 MAY 1977 



as the presentation of lesson objectives could not I . evaluated in the current study 
since all of the lessons sampled contained such objectives. Similarly, the numbers 
of embedded questions and lesson test items were found to be so highly correlated 
with the amount of lesson content that it was necessary to exclude them from the 
predictor set. Relatively little of their variability was independent of lesson 
length. While the post-instructional organizer, Number of Self-Check Ite ms, was 
retained in the predictor set, the typical number of such items per lesson was so 
small that one should be hesitant in using the regression equation information to 
evaluate the utility of such organizers. It would be desirable to samole a much 
wider variety of values of course ontent variables in any future efforts at model 
refinement. The current study's limitation to two courses makes it difficult to 
disentangle the effects due to differences in course content from those due to 
unmeasured, course-specific factors. Finally, it would be highly desirable if 
future work could include examination of a variety of alternative instructional 
media, including computer.- assisted instruction. 

The final data problem area concerns the intercorrelation of predictor variables 
If it were possible to manipulate directly the variables of interest, one would 
impose experimental controls to hold other conditions constant while varying one, or 
at most a limited set, of variables. Since this was not possible, the current study 
employed a regression analysis approach in an attempt to examine statistically the. 
effects of individual variables. The regression approach is limited, however, in 
that the confounding effects due to the intercorrelations among varidbles cannot be 
controlled. For example, if a particular instructional method wa^ dlwdys employed 
for teaching a particular class of content, and never employed for teaching other 
categories of content, its effect could never be evaluated. Nor could one determine 
the expected time savings due to self-pacing with other instructional methods for 
that class of content. Other, less extreme examples can also present problems. 
Within the current AIS systeiii, the more expensive audio-visual media are generally 
employed only for those lessons which experience has proven to be particularly diffi- 
cult for students when the material was presented in the less expensive, pruy rainnied 
text format. Thus, a regression equation predicting lesson cumpletion tine which 
incorporated a presentation medium variable would, under these circumstances, tend 
to misrepresent the utility of audio-visual presentation as coinpared tc programmed 
text because data on the audio-visual mode were not sampled across tfie full range of 
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lesson content. It is anticipated that problems of this type will continue to be a 
major limitation to further development of the predictive model. 

The computer cost analysis model developed in this study incorporates an 
estimate of students' course completion time based on relevant course content, 
instructional design, and student characteristic variables. It thus increases the 
scope of Rand's MODIA cost model by basing course completion times on predicted 
values from a regression equation derived from actual data from an operational 
environment. The flexibility of input preparation allows the model to be used for 
both general and detailed levels of analysis. 

The cost figures produced by the program are only as good as the accuracy of 
the course description provided and the similarity of the course to be modeled to 
the AIS courses from which data were sampled in this study. A further limitation 
of the cost analysis program is that only mean cost values are produced even though 
considerable variability is associated with the predicted course completion times. 
Further work on the AIDE cost model could improve its usefulness by computing the 
variances associated with the final course cost. The inclusion of variances would 
allow analysis of variance and linear model comparisons to be made on the costs of 
alternative approaches. As the model now stands, exercising it with respect to 
.uestions related to course content, in'':>tructional design, arid student character- 
istics produces cost figures which can only be compared in terms of percentage of 
change without any measure of the significance of the cost differences. 

In summary, the current study has provided three '^^oducts which can facilitate 
the investigation of cost savings due to the individualization of milita y training. 
The Descriptive Model of Computer-Based Training Systems will, hopefully, provide a 
context within which to investigate the various components of such systems. The 
Predictive Model has explored a methodology which, with minor variations, can be 
developed to predict timesavi igc due' to individualization accurately in a variety of 
training environments. As instructional syst?ms such as the AIS be:ome established, 
a broad, reliable data base will become available which can be used to support a 
systematic, empirical approach t the selection of additional courses for individual- 
ization. Finally, the AIDE Cost Modei has extended the Rand MODCOM program to 
incorporate the variable course completion times characteristic of computer-based 
training systenis. 
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One final comment would appear to be appropriate. As was discussed 
in Section 5.3 of this report, Discussion of AIDE Example Run Results , those 
training expenses which are directly attributable to students and which are 
sensitive to variations in course completion time, (i.e., student pay and 
allowances) form a relatively small proportion of total course costs. If 
substantive cost savings are to be realized from the individualization of 
instruction, the total training system, including support personnel and 
facilities, must be critically examined. For example, the emphasis to 
date has been on shortening course completion time while holding the number 
of instructional personnel constant. It may well be more cost effective to 
allow somewhat longer course completion times^while reducing the student/ 
instructor ratio. The coordination of student arrivals and departures from 
the training base is at least as important as is the efficient use of the 
student's time in the classroom. Personnel whose duties are only tangentially 
related to training account for a sizeable portion of a training base's 
budget. Reducing the length of one or two small enrollment courses by 20 
or even 30 percent will not appreciably decrease the number of students 
on base and will not, consequently, allow any reductions in the number 
of these support personnel. If, on the other hand, a substantial proportion 
of the training on a given base were individualized and the expected time 
avings realized', a number of indirect costs could be reduced. Referring 
the functional cost items shown for the standard sample in Table 7, one 
vouiH expect broad scale reductions in course times to impact the following 
items: Facility Construction; Instructor Pay and Al lowances ; Facilities 
Maintenance Personnel; Trainiro Administration Personnel; Base Operating 
Support Personnel; Medical Personnel, PCS costs for instructors; and Instructor 
Education Training. These items account for an additional 21 percent of the 
total course costs, and reductions in these areas would almost certainly 
result in substantive savings. It may well be the case that the full potential 
of compute's-based ti^aining systems for cost reduction will only be realized 
when they have become the norm rather than the exception in military training. 



^6-8 



EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUME II 



MDC El 570 
15 MAY 1977 



7.0 REFERENCES 



Asbury, C. A. Selected factors influencing over- and under-achievement in young 
school age children. Review of Educational Research , 1974, 44(4), 409-428. 

Bern, D. J. and Allen, A. On predicting some of the people some of the time: The 
search for cross-situational consistencies in behavior. Psychological Review, 
1974, 81(6), 506-520. 

Brown, B. R. and James, T. G. An investigation of performance contingent presenta- 
tion rate . Paper prepared for the annual meeting of the American Psycholoaical 
/Association, Honolulu, September 1972. 

Campbell, D. S. and Schwen, T. M. Beyond the remedial loop: Toward an integration 
of task and learner analysis for a process approach to instructional development . 
Papp*- prepared for the annual meeting of the Association for Educational Communi- 
cations and Technology, Philadelphia, March 1971. 

Carroll, J. B. The prediction of success in intensive foreign language training. 
In 1. Glaser (Ed .) Training research and education . Pittsburgh: University 
of .Pittsburgh Press, 1962. 

Carroll, J. B. A model of school learning. Teachers Col lege Record, 1963, 62, 
723-733. ^ " 

Col ton, F. V. Effects of giving students data on task completion time in a college 
media course. Audio-Visual Communications Review , 1974, 22^(3), 279-294. 

Deignan, G. M. Development of the Delta Reading Vocabulary Test (Professional 
Paper). Lowry Air Force Base, Colorado: USAF Human Resources Laboratory, 1973. 

Deignan, G. M. Delta training aptitude tests: Biographical > attitude, and 

concealed figures tests . Lowry Air Force Base, Colorado: USAF Human Resources 
Laboratory, 1974. 

Glaser, R. Components of a psychology of instruction. Review of Educational 
Research, 1976, 46, 1-23. 

Guilford, J. P, The Nature of Human Intelligence . New York: McGraw-Hill, 1967. 

Hess, R. and Kantar, P. MODIA: The cost model (WN-9044-PE) . Santa Monica: Rand 
Corporation, June 1976. 

Kerlinger, F. N. and Pedhazer, E. Multiple Regression in Behavioral Research . 
New York: Holt, Rinehart, and Winston, 1973. 

Leherissey (McCombs), B. L. The development of a measure of siate epistemic 

curiosity (Tech. Memo No. ITT Tallahassee, Florida: Florida State University, 
May 1971. 

McAyoy, R., Kalin, M. , and Franklin, T. An analysis of personality variables in 
individual modes of instruction . Paper prepared for the annua i meeting of the 
American Educational Research Association, New Orleans, Louisiana: February 1973. 

94 ■ 



EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUME II 



MDC El 570 
15 MAY 1977 



McCombs, B. L. The development and validation, of (.ourse-speci f ic reading skill 
scales for Air Force technical training (Unpublished working paper), Lowry 
Air Force Base, Colorado: McDonnell Douglas Corporation, August 1974. 

McCombs, B. L. The development and validation of general and specifi c media 
preference scales for use in an adaptive instructional system (Unpublished 
working paper). Lowry Air Force Base, Colorado: McDonnell Do'jglab Corporation, 
October 1974. 

McCombs, B. L. and Siering, G. The design and development of an adaptive ro de l for 
the Air Force advanced i nstructional system . Paper prepared for the 6th congrpss 
of the International Ergonomics Association. College Park, Maryland: July M ^. 

Merrill, M. D. and Boutwell, R. C. Instructional development: Methodology and 
research. In Kerlinger, F. N. Review of Research in Education . Itasca, 
Illinois: F. E. Peacock, Inc., 1973. 

Nie, N. H., Hull, C. H., Jenkins, J. G., Steinbrenner, K., and Bent, D. H. Statis- 
tical Package for the Socia l Sciences (2nd ed.). New York: McGraw-Hill, 1975. 

Rhetts, J. E. Task, learner, and treatment variables in instructional design. 
Journal of Educational Psychology , 1974, 66(3), 339-347. 

Rockway, M. R. and Yasutake, J. Y. The evolution of the Air Force advanced 

instructional system. Journ al o f Educational Technology Systems, 2(3), Winter 
1974^ 

Rotter, J. B. Generalized expectancies for internal versus external control of 
reinforcement. Psychological Monographs , 1966, 80, Whole No. 609. 

Sarason, I.G. Effects on verbal learning of anxiety, .cassurance, and meaningful- 
ness of material. Jou^^nal of Experimental Psycholog y, 1958, 56, 472-477. 

Schwen, T. M. Learner analysis: Some process and content concerns. Ajjdiu-Visual 
Co mmunicat-^ons Review , 1973, 21(1 ), 44-72. 

Spielberger, C. D. Theory and research on anxiety. In C. D. Spielberger (Ed.), 
Anxiety and Behavior . New York: Academic Press, 1966, 3-20. 

Spielberger, C. D., Gorsuch, R. L., and Lushene, R. E. Manual for the State-Trait 
Anxiety Inventory . Palo Alto, California: Consulting P'-vchologist Press, 1970. 

Spielberger, C. D., O'NeiK H. F. , and Hansen. D. N. Anxiety, drive theory, and 
computer-assisted learning. In B. A. Maher (Ed.), Progress in experimental 
personality research: VI . New York: Academic Press, 1972. 

Uprichard, A. E. Variables to consider in planning research ^or effective instruc- 
tion: A conceptual framework. Educational Technology , 1975, 15(7), 29-32. 

U.S. Army Training and Doctrine Command. Interservice Procedures for Instructiona l 
Systems Development 95 volumes) (TRACOC Pamphlet 350-30) . Fort Benning, Georgis: 
Combat Arms Training Board, August 1975. 



ERIC 



7.2 95 

MCOONNCLL aaUCLAS AST99<y/%/AUTtCS CO/Vff'A/VV- EASY 



EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUME 11 



MDC El 570 
15 MAY 1977 



Wagner, H., behringer, R. D., and ^ittie, C. L. Individual course completion time 
pr edictions: Development of instruments and techniques (HumRRO Technical Report 
73-23). Alexandria/ Virginia: Human Resources Research Organization, November 
1973. 

Walberg, H. J. Learning models and learning environments. Educational Psychologist , 
1974, 11(2), 102-109. : 

Ward, J. H., Jr. and Jennings, E. Introduction to Linear Models . Englewood Cliffs, 
N^w Jersey: Prentice-Hall, 1973. 

Yasutake, J. Y. and Stobie, W. H. The Air Force advanced instructional system (AIS): 
A n overview . Paper prepared for the American Educational Research Association, 
Chicago, Illinois: April 1975. 



f^coaryfSiCLL noi/ctAS astro^n/xutbch caMMfKv ^ fast 



EVALUATING 

INSTRUCTIONAL SYSTE^^S VOLUME II 



APPENDIX A 

AID FOR INSTRUCTIONAL DEVELOPMENT AND EVALUATION 



A-1 97 

MCoo,yryei.i. oouolas astronautics coi>ff»AivY • ea 



EVALUATING 
INSTRUaiONAL SYSTEMS 



VOLUME II 



MDC El 570 
15 MAY 1977 



GENERAL 

This appendix describes some technical aspects of the AIDE computer cost 
.analysis p'^ogram. Appendix B gives a complete FORTRAN listing of the AIDE program. 
Appendix A includes the following: 

0 Computer Program Overview 

0 Differences Between AIDE and MODCOM Data Inputs 

0 Using a different Regression Equation 

0 General Comments About the FORTRAN Program 

0 Input Data Preparation 

0 Computer Program Output Description. 



ERIC 



98 

A-2 

MCDONNKt.L. OOVOLA9 Amr/tOMAUTtCS COMPAMY m mAmr 



EVALUATING 
INSTRUCTIONAL SYSTEMS 



VOLUMf II 



MDC El 570 
15 MAY 1977 



Computer Program Overview 

The computer program was called Aid for Instructional Development and Evalua- 
tion (AIDE). The development of the AIDE computer program was facilitated by split- 
ting the program into the following major modules: 

Input Module : Read and check input data.. Input data will include lesson 
descriptors, student descriptors, instructor data, courseware data, instruc- 
tional resources data, physical facilities data, and CMI system data. A com- 
plete list of the input data is given in a subsequent section of this Appendix. 

Course Duration Module : Use the regression model and overhead factor developed 
in Section 4.0 to compute the average course duration and the percentage time- 
saving. 

Student Flow Modul e; Use the average time-to-complete the course to compute 
the average enrol l.'nent level in the course. 

Resources and Personnel Module ; Based on the average enrollment in the course 

c 

and other user supplied course data, find the courseware development and main- 
tenance manhours, student and instructor staff manhours, instructional hard- 
ware and other resources procurement quantities and maintenance manhours, and 
computer system hardware quantities. In determining these quantities, allow- 
'ance is made for fluctuations in the enrollment level. 

Co5t Accumulation Module : Combine personnel and resource quantities with unit 
costs to arrive at development, initial investment, and annual operations and 
maintenance costs to operate the course over a period of five years. 

Output Module : Write intermediate and final results. 

Prior to the start of programming the model, our literature survey and contacts 
with instructional system developers revealed that the. Rand Corporation had been 
;developing an instructional model called MODIA (Methods of Designing Instructional 
Alternatives). The MODIA Cost Model (MODCOM) was well developed and performed 
several of the functions needed in the AIDE Model. Specifically, these are: 

0 Several input routines for courseware, hardware and, personnel 

0 A routine to generate student manyears, given the average time to complete 
the course 
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0 Estimation of training administration, base operating support, and medical 
support personnel 

0 A cost accumulation subroutine 

0 Routines to summarize input data, intermediate results, and final results. 

In view 0/ these similarities, we requested and received a Qopy of the MODCOM 
computer program from the Rand Corporat/on. 

I • 
The programming efforts were then directed at writing the subroutines to compute 

the course duration and estimate resource quantities and modifying the MODCOM program 

so as to iritegrate it with these subroutines. A listing of the. computer program is 

given in Appendix B. The inputs tp^ be supplied by the user and the outputs of the 

computer program, are described in the following paragraphs. 

Differences Between AIDE and MODCOM Data Inputs 

The^first 15 data card types of AIDE are similar to thoseoOf MODCOM, but AIDE 
adds a 16th data card type for lesson descriptors,. Specific format differences 
exist in AIDE for the following card types: 3, 9, 10 and 12. AIDE always uses 
the calculation option where MODCOM gives a choice of calculation or throughput. 

Using a Different Regression Equation 

A regression equation of u| to three student varibables and up to eleven 
lesson variables can be entered by changing the DATA statement values in lines 
91 through 98 of the program AIDE with no othf.'r reprogramming necessary. NSVAR 
and LDES are the number of student and lesson variables respectively. TVAL{11) 
is the vector of maximum values for the lessen variables. Regression equations 
of a different combination of variables wil| require additional changes in the 
FORTRAN program in PROGRAM AIDE and in subroutines (overlays) READS, READW, AIDEA 
and AIDEB. Specifically, the common statement of AIDEIA and AIDE2 will be involved 
as well as read and write formats in PROGRAM AIDE and subroutines READS, READW, 
AIDEA and AIDEB. J 
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General Comments About the FORTRAN Program 

The AIDE FORTRAN Program was developed on the CDC Cyber 73-16 computer at AIS, 
Lowry Air Force Base, Colorado. The program is presently overlayed to fit within 
a 130K core memory limit. The overlay will need to be re-setup by a programmer for 
the local computer facility. The flowcharts and other detailed documentation 
accompanying the MODCOH FORTRAN program (Hess and Kantar, 1976) apply to AIDE 
as well, with the exception that 'AIDE adds three subroutines (READW, AIDEA, and 
AIDEB) and deletes one subroutine (OUTS). 

Input Data Preparation \ 

This section describes the inputs to the a|[dE computer program. The general 
form and style of the inputs are derived from the Rand MODCOM program which played 
a large part in the development of our AIDE model. The description of the Rand 
model variables is taken from a report provided to us prior to publication by Rand. 

A large amount of data can be entered into the program. Only data relating to 
lessons (regression model variables), student variable scores, and class entry, inter- 
val are required. Any other data omitted will be treated either as if it wereizero 

i 

or assigned a default value. 

Data are input to the model on cards. Sixteen types of card formats are used, ai 
listed in Table 11. The detailed form of each card is described below. The follow- 
ing general comments apply to all cards: 

Format Identifier : All cards must have the format type punched in columns 
1 and 2. 

Blanks : Unless otherwise specified, all blanks in numeric fields are read as 
zeros . 

Mandatory Inputs : The variables which must be assigned non-zero inputs are: 
"entry interval (in hours)," on card format 2; student variables scores on 
card 3; and at least one set of lesson descriptors (card 16). Failure to 
enter a non-zero value for these variables will result in the printing of an 
error message and program termination. Format type 99 is used to signify the 
end of the input data; one card of this type must be present. Failure to 
enter this card will result in a FORTRAN error rassage and program termination. 
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TABLE 11 




DATA CARD TYPES 


mmj NUMBER 


CARD TYPES 


1 

1 




2 


i^onoy^sl P/^iiv^cA Cr>^/^^/\w>»r* 


3 


^"huHpiTh Tnnii'hc 

o tuucii u 1 npu Ub ^ 


4 


Tnc■^r'll^"^^r' Tnmi'hc 


5 


LrUUrbcWarc IripUUb 


6 


LrUi r iLUiuMi naiipuwcr inpUuS 


7 


Hardware Procurement Inputs 


8 


Hardware Maintenance Manpower Inputs 


9 


Facility Procurement Inputs^ 


10 


Facility Maintenance Manpower Inputs 


11 


Training Administrative, Base Operating Support, and Medical 
Manpower Inputs' 


12 


computer Management System Inputs 


13 ' 


Optional Officer/Airman/Civilian Distribution Overrides 


14 


Optional Miscellaneous Overrides 


15 


Optional ^Pay and Allowance Overrides 


16 


Lesson Descriptors 


• 99 


Termination Card 



Card Limit: In order to preserve core memory, it was nVcessary to put limits 
on the number of cards of each format t. ^e which can entered for a single 
run of the AIDE Model. This limit is indicated in the description of each 
card type. Cards in excess of that limit are deleted' from further processing. 
Card Deletions: Throughout this section, mention is'^made of card deletions 
which will occur if certain input conditions are violated (e.g., input not 
within acceptable range, and illegal input combinations). All such card 
deletions will be listed as part of the output with an appropriate error message, 
Decimal Poin^: All input data are entered as real (floating-point) numbers. 
The implied decimal point location is at the right end of each field unless 
otherwise specified. However, a punched decimal point always overrides an 
implied decimal point location. 
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F orniat 1: Title Card Card Limit = 1 - This card is used to identify AIDE runs. 
The title, to be entered in columns 3 through 80, may contain between 1 and 78 
alphanumeric characters. If this card is excluded, then the default title, AIDE 
model , is used. 

Format 2: General Course Factors Card Limit = 1 - This card contains inputs, 
describing washouts, instructor training costs, and :-he old course duration. 

Columns 3-7 (5 column numeric field): Old course duration (in hours). This 
number, as described in Section 4.2, is the number of hours of instruction 
presented in the old course. If a value is not entered, the new course 
duration will be computed, but not time savings. 

Columns 8-11 (4 column numeric field): Washout rate (%): The ratio of the 
number of washouts to the total number of entrants times 100. 

Columns 12-15 (4 column numeric' fipl d) : Entry interval (in hours): The 
average length of time separating entry classes. 

Columns 16^19^ (4 column numeric field): Number of months instructors on board 
in year 0: The number of months the initial instructor cadre will spend pre- 
paring for the subject course in year 0. This value, which must be supplied 
by the planner, should include time required for special factory training and 
education training as well as time necessary for curriculum development. 

Columns 20-24 (5 column numeric field): Cost of initial factory training (in 
dollars): The total cost, less student pay and allowances, of providing the 
initial instructor cadre with specialized equipment or system training neces- 
sary for cQurse development and conduct. This initial specialized training is 
usually accomplished at a private contractor's facility. Subsequent specialized 
training of instructors is usually provided informally by the existing instruc- 
tor nucleus. This cost should ir.ciude travel per diem, transportation expenses, 
destination per diem, and attendance fees (or contract ccsts), as applicable. 
The pay and allowances of the initial instructor cadre while in student status 
will be accounted for elsewhere. 

Columns 25-29 (5 column numeric field): Cost of instructor education training 
($): The cost of the six-week instructor prerequisite course on education 
principles. 
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Format 3: Student Inputs Card Limit - 75 - This card contains inputs relating 
to the number and type of student entrants, and values of the three student charac- 
teristics variables (ASVADM, ASVMEC, SEX). 

Breakdown of Student Entrants - The course planner must determine the breakdown 
of total annual student entrants by personnel type (Active Duty Force Pipeline, 
Active Duty Force Lateral/Upgrade, Guard/Reserve Pipeline, Guard/Reserve Lateral/ 
Upgrade, Other DoD Pipeline, Other DoD Lateral /Upgrade, Non-DoD), personnel designator 
(officer, airman, civilian), and paygrade. Typical paygrades for Air Force pipeline 
and lateral /upgrade students are: 

Student Type Offiter Airman 

Pipeline 0-1 / E - 1 

Lateral or Upgrade 0-2 thru 0-4 E - 3 thru E - 5 

Columns 3-4 (2 column numeric field): Personnel Type: Numeric personnel type 
' code as follows: 

01 = .'^ctive Duty Force Pipeline Student 

02 = Active Duty Force Lateral Upgrade 

03 = Guard/Reserve Pipeline Student 

04 = Guard/Reserve Lateral Upgrade Student 

05 = Other DoD (Army, Navy) Pipeline Student 

06 = Other DoD (Army, Navy) Lateral Upgrade Student 

07 = Non-DoD (MAP) Student 

Use of a code outside the range 1 through 7 will result in the deletion of the 
card. 

Columns 5-6 (2 column numeric fiel d) : Personnel Designator: Numeric personnel 
designator code as follows: 

1 = Officer 

2 = Airman 

3'= Civilian-GS (General Schedule) 

4 = Civil ian-WB (Wage Board) 

5 = Non-DoD 

Use of a code outside the range 1 through 5 will result in the deletion of that 
card* Additionally, card deletion will occur if personnel designator 5 is used 
in combination with any personnel type other than 7. 
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Columns 7-8 (2 column numeric field): Pay Grade: Standard U.S. Government 
numeric pay grade codes. The code entered in these columns must be within the 
range of permissible paygrades for the specified personnel designator or else 
the card will be deleted. A duplicate persor.nel type/personnel designator/ 
paygrade combination will also cause a card to be deleted. The number of 
unique paygrades entered for'a given persoi , ,1 type/personnel designator com- 
bination will depend on the required level of accuracy and the ambition of the 
user: each paygrade requires a separate card entry. ATC generally uses a 
single paygrade (the modal) for each personnel type/persor.nel designator com- 
bination. In order to reduce the planner's input requirements, pay and 
allowance factors for each paygrade have been stored as part of the model. 

Civilian (Personnel Designator - 3 and 4) 

Stored P&A Factor ($) Stored P&A Factor ($) 



Paygrade for GS Civilians (3) for WB Civilians (4) 

18 39,060 

17 39,060 

16 39,060 

15 35,823 

14 30,377 15,157 

13 28,053 14,773 

12 22,285 13,845 

11 18,935 13., 373 

10 17,041 12,758 

9 15,717 12,127 

8 14,254 11,471 

7 13,828 11,077 

6 11,782 10,527 

5 10,483 9,587 

4 9,266 9,214 

3 7,953 8,896 

2 6,447 8,491 

1 5,651 7,912 

Blank or 0 12,599 ' 11,464 
(Overall 
Average) 



Additionally, a dummy pay and allowance schedule has been set up for non-DoD 
students with 20 possible paygrades. However, because of the wide range of possible 
pay rates, typical values could not be established; and consequently, the stored 
values were set equal to zero. If the planner feels that non-DoD student pay and 
allowances are relevant to a particular costing exercise, then the pay factor over- 
ride option should be employed to introduce the appropriate pay rate. 

o . 3.05 
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Columns 9-11 (3 column numeric field): The score of this type of student on 
the ASVADM test (0-100). 

Columns 1^-14 (3 column numberic field): The score of this type of student on 
the ASVMEC test (0-100). 

Columns 15-17 (3 column numeric field): The ratio of male students to the 
sum of male and female students (0-1.0). 

Columns 18-37 (five 4-co1umn fields): Number of student entrants of this type 
■ in years 1-5. 

Format 4: Instructor Inputs Card Limit = 25 - This card contains inputs 
pertaining to the size, turnover, and student loading of the instructor force. 

Breakdown of Instructor Force - The course planner must determine the breakdown 
of total instructor force by personnel type (Air Force, Other DoD), personnel desig- 
nator (officer, airman, civilian) and paygrade, and student load. Typical paygrades 
for each instructor type are provided in Table 12. 



TABLE 12 
TYPICAL INSTRUCTOR PAYGRAOES 



Instructor Type 


Officer 


Airman 


Civilian 


Academic, remedial or special 
requirements instructor 

Evaluators and Supervisors 


0 - 3 
0 - 4 


3 - 5 
E - 6, E - 7 


GS - 9 
GS - 11 



Columns 3-4 (2 column numeric field): Personnel Type: Numeric personnel type 
code as follows: 

8 = Air Force Instructor 

9 = Other DoD (Army, Navy) Instructor \ 
Use of a code other than 8 or 9 will result in the deletion of that card. 
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Colunns 5-6 (2 column numeric field): Personnel Designator: Numeric personnel 
designator code as follows: 

1 = Officer 

2 = Airman 

3 = Civilian - GS 

4 = Civilian - WB 

Use of a code outside the range 1 through 4 will result in the deletion of 
that card. 

Columns 7-8 (2 column numeric field): Paygrade: Standard U.S. Government 
numeric paygrad codes. The codes and stored pay factors are the same as 
those described for input format 3. As before, the code entered in these 
columns must be within the range of permissible paygrades for the specific 
personnel designator or else the card will be deleted. A duplicate personnel 
type/personnel designator/paygrade combination will also cause a card to be 
deleted. 

Columns 9-11 (3 column numeric field): Turnover rate (%): The percentage of 
instructors who will be replaced each year or the reciprocal of the average 
instructor's tour length in years times 100. 

Columns 12-16 (5 column numeric field): Size of initial instructor cadre in 
year 0; the number of instructors (not manyears) brought onboard in year 0 for 
curriculum development and any required factory training. 

Columns 17-19 (3 column numeric field) :o Students per Instructor: The nurriber 
of students that can be instructed or counseled by an instructor of this type. 

Format S: Courseware Procurement Inputs Card Limit = 75 - The model recognizes 
four classes of courseware: ' (1) printed media such as textbooks and workbooks which 
are intended for both stud'^nts and instructors; (2) printed media such as lesson and 
evaluation guides which are intended only for instructors; (3) display media such as 
films, slides, and charts; and (4) software such as teaching machine program<^. 

Four separate costs are estimated: (1) The initial cost of producing the master 
from which all subsequent copies will be made; (2) the cost of reproducing and pack- 
aging the required number of copies; (3) the "rmual cost of revision due to change 
in course content; md (4) the annua, cost of rr.placement due to loss, damage, or 
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normal aging The cost of courseware development, such as script writing and edit- 
ing, is accounted for within curriculum manpower pay and allowances. 

Columns 3-4 (2 column numeric field): Courseware ID Number: Unique number for 
each courseware type in range 1 through 99. A carcl will be deleted if its ID 
number duplicates that of another card or if its ID number fal is outside the 
prescribed range. ' 

Column 5 (1 column numeric field): Class: Numeric courseware class code as 
fol lows: 

1 = Printed media for students and instructor^ (texts, workbook) 

2 = Printed media for instructors only (lesson and evaluation guided 

3 = Display media (films, slides) 

4 = Software (teaching machine programs) 

Use of a code outside the range 1 through 4 will result in the deletion of that 
card. 

Columns 6-27 (22 column alphanumeric field): Name of courseware: User- 
selected name for identifying each courseware type. 

Columns 28-31 (4 column alphanumeric field): Name of courseware copies: 
User-selected name for identifying the unit denominator (book, reel, set) of 
each courseware type. 

Columns 32- 37 (6 column alphanumeric field): Name of courseware measure: 
User-selected name for identifying the unit of measure (pages, minutes, slides, 
etc.) for each courseware type. 

Columns 38-41 (4 column numeric field): Total number of courseware measure per 
copy: The number of pages, slides, or minutes, etc., per unit of courseware. 

Col umns 42-46 (5 column numeric field): Initial preparation cost per course- 
ware measure ($): The cost of producing a single measure of master courseware. 
Some typical values are provided in Table 13. 

Columns 47-50 (4 column numeric field): Copy cost per courseware ^measure ($): 
The cost of copying a single measure of master courseware. .Typical values are 
again provided in Table 13. 



A.12 ^'08 \ 
gj^^" f%^oaasj/M£LL aauaLAS Asr-ffc^fs/Auncs caMMNv m east 



o MDC El 570 

VOLUME li 15 MAY 1977 

Columns 51-54 (4 column numeric field): Packaging cost per unit: The cost of 
cassettes, reels, trays, etc, per unit of courseware. Some typical values 
are as follows: 



Sound on Slide Tray 


$30.00 


Slide Tray - 40 slides, straight 


2,50 


Audiotape Cassette 


2.00 


Video r ^ Tape {ij minutes) 


18,00 


Video Cassette Tape (60 minutes) 


25.00 


Loose-leaf Binder (3-12", 
Telescoping Poles) 


3.00 



TABLE 13 

TYPICAL COURSEWARE PROCUREMENT COSTS 





Courseware 
Measure 


Initial 
P; oaration Cost/ 
Courseware Measure 
Dollars 


Copy Cost/ 
Coui are Measure 
uoTlars 


Printed Text 


Pages 


$1.50 - $4.50 


.01 


Sound Slide Set (Color) 


Slides 


50- 1,00 


.08 


35imi Color Slide (Text) 


'Slides 


4.50 - 9.00 


.0.8 


35mm Color Slide (Illustration) 


Slides 


7.50 - 25.00^ 


.08 


8mm Animated Sequence 


Seconds 


•40 - ,85 




Video Tape 


Minutes 


1.00 - 2.00 





Columns 55-57 (3 column numeric field): Annual revision rate (%): The. per- 
centage of each courseware type's actual presentation minutes which can be 
expected to be revised annually due to changes in course content. These 
changes in course content could be caused by such things as modifications to 
mission equipment, a required upgrading of student capabilities, or an effort 
to improve the utilization of course resources. The model assumes that after 
the necessary revisions are made to the master, sufficient copies will be made 
of the revised portions (only) and integrated into the existing copies." 

Columns 58-60 (3 column numeric field): Annual replacement rate (%): The 
percentage of each courseware type's units which will need to be replaced 

annually due to damage or normal aging. 

1 
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Columns 61-64 (4 column numeric field): The number of lessons in the course 
^ utilizing this courseware. Courseware description need not limit it to only 
one lesson. ^*Fbr instance, several programmed text lessons could be grouped 
into one data entry. If blank, value is set equal to one. 

Columns 65-67 (3 column numeric field): Number of simultaneous users per copy. 
This entry is used for courseware that is used simultaneously by more than one 
student, as in a four-man AV presentation. If blank, value is set equal to one. 

Format 6; Curriculum Manpower Inputs Card Limit = 75 - Curriculum personnel 
-determine training requirements and develop course written material. Normally,^ 
attached to the Curriculum Unit of a training department, these personnel share the 
task of courseware development with^ the course's instructors. 

Columns 3-4 (2 column numeric field): Courseware ID number: The number used, 
to identify the same courseware type in. input format 5. A format 6 card will 
be deleted if its ID number does not have a match on a f-ormat 5 card, or if 
its ID number duplicates that of another format 6 card. 

Columns 5-9 (5 column numeric field): Number of course h^urs: The number of 
course hours taken up by a given courseware type. This number should be 
coordinated with the number of lessons specified on card type 5 for this type 
of material . 

Columns 10-14 (5 column numeric field): Number of curriculum manho.urs per 
course hour: The number of initial development curriculum manhours required 
for each course hour. Generally speaking, this number is a function of the 
teaching agent and the teaching format and is independent of the courseware type. 
Typical values are provided in Table 14. 

Columns 15-47 (3 column numeric field): Percentage accomplished by instructors: 
The percentage of curriculum development done by the instructors rather than 
curricul um personnel . 

^o*"™^ 7: Hardware Procurement Inputs Card Limit = 75 - The model estimates 

four types of hardware costs: (1) the cost of purchasing'the incremental number of 
units required each year, including initial replacement part stocks; (2) credits for 
any surplus items returned to the Stock Fund; (3) the annual cost jf "replacing those 
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TABLE 14 

INITIAL COURSEWARE DEVELOPMENT 
MANHOURS PER CONVENTIONAL COURSE HOUR 



Presentation 
Media 


Development* 
Manhours 'per Hour 


PrOflrammpd Tpvt 

c 


bU - 70 


Audio Visual 


80 - 100 


Computer Assisted Instruction 


100 - 300 


*If new material is being written 
notes from the old course, these 
increased by about 40%. 


, without benefit of 
nuiiibers should be 







units worn out during the year; and (4) the annual cost of misecl laneous repair 
parts. 

Columns 3-4 (2 column numeric field): Hardware ID number: Urn'que number for 
each hardware type in range 1 through 99. A card will be deleted if its ID 
number duplicates that of another card if its ID number falls outside the 
prescribed range. 

Column 5 (1 column numeric field): Class: Numeric hardware class code as 
follows: 

1 = media hardware (slide projectors, reel-to-reel projectors, tape 

recorders, teaching machines) 

2 = special equipment (trainers, balled aircraft) 

3 = overhead hardware (chairs, desks) 

Use of a code outside the range 1 through 3 will result in the deletion of '"that 
card. 

Columns 6-35 (30 column alphanumeric fie-ld): Name of hardware type: User- 
selected name for identifying each hardware type. 

Columns 36-39 (4 column numeric field): The number of units of a given hard- 
ware type which are available to a course at its inception, at zero cost. ' 

Columns 40-47 (8 column numeric field): Procurement cost per unit ($). 
Normally, an AF planner will consult his own service's supply manual to 
determine hardware unit costs. 
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Column 48 (1 column numeric field): Credit Option: Option to obtain credit 
for surplus equipment returned to Stock Fund (yes = 1; no = 0 or blank). The 
planner may obtain credit for surplus equipment returned to the Stock Fund 
provided both^the following conditions are met: 

a. The unit cost does not exceed $1,000. 

b. A demand exists elsewhere in the AF for the returned items. 

The option should not be exercised if either the unit,cost exceeds $1 ,000 or 
if other AF demand does not exist. In fact, if the planner enters a "1" for 
an item costing more than $1,000, then the model automatically resets the ' 
option designator value to "0." The planner should also enter a zero if any 
surpluses generated in a given year are to be retained for increased demands 
in subsequent years. The amount of the credit is 60% of the original purchase , 
price, the other 40% assumed to go into rehabilitation action so ,that the 
equipment may again be sold at the full purchase price. 

Columns 49-51 (3 column numeric field): Annual attrition rate (%): The ; -r- 
centage of hardware units which will need to be replaced each year. This value 
can be approximated by multiplying the reciprucal of the estimated average life 
(in years) by 100. , 

C olumns 52-57 (6 column numeric field): Miscellaneous repair part cost per 
unit per year ($): The annual cost of purchasing repair parts for a single 
unit. In the absence of better data, a value .in the range of 6 to 10% of the 
unit procurement cost should be used. 

Columns 58-61 (4 column numeric field): Number of lessons using this type of 
hardware. More than one lesson may utilize the same type of hardware (such as 
an AV projector); this item allows the planner to specify this fact. 

Columns 62-64 (3 column numeric field): The number of students who may simul- 
taneously use hardware item. This is the number of students who are studying 
the same lesson that may simultaneously use this hardware device. 



Format 8: Hardware Maintenance Manpower Inputs Card Limit = 75 - Hardware 
maintenance personnel are responsible for the preventive and corrective maintenance 
of course hardware. Media hardware maintenance is generally the responsibility of 
the Training Services Division (school-level), while all other types of hardware are 
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usually serviced by the base Maintenance and Supply Group or a specialized mainte- 
nance squadron. 

Columns 3-4 (2 column numeric field): Hardware ID number: The number used to 
identify the same hardware type in input format 7. A format 8 card will be 
deleted if its ID number does not have a match on a format 7 card or if its 
ID number duplicates that of another format 8 card. 

Columns 5-10 (6 column numeric field): Number of failures per hour of use: 
The average number of times a particular type of equipment can be expected to 
fail for each hour of use. > 

Columns 11-15 (5 column numeric field): Average time to repair per failure 
(in hours). This value should include only that portion of downtime in which 
maintenance personnel are actively pursuing the repair activity. 
«. 

Format 9: Facility Procurement Inputs Card Limit = 25 - This card contains 
inputs relating to the modification and construction of course facilities. 

Columns 3-4 (2 column numeric field): Facility ID number: Unique number for 
each facility type in range 1 through 99. A card will be deleted if its ID 
number duplicates that of another card or falls outside the prescribed range. 

Columns 5-34 (30 column alphanumeric field): Name of facility type: User- 
selected name for identifying each facility type. 

Columns 35-38 (4 column numeric-field): Units available "as is" at start of 
first-year: The number of units of a given facility type which are available 
to a course at its inception at zero cost. 

Columns 39-45 (7 column numeric field): Construction cost per unit ($): The 
cost of constructing or modifying a single facility unit. 

NOTE : If unit requirements for a given facility type are assumed to be avail- 
able at zero cost for all years under study, then the user should enter 
a unit construction cost of zero. 

Columns 46-49 (4 column numeric field): The number of students who can simul- 
taneously use the facility. An example is the number of students who can 
simultaneously use a laboratory. 
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Format 10; Facility Manpower Inputs Card Limit = 25 - Facility maintenance 
personnel are responsible for the maintenance and repair (including custodial 
service) of course facilities. 

Columns 3-4 (2 colurrn numeric field): Facility ID numbrr: The number used to 
identify the same .facility type in input fori;iat 9. A format 10 card will be 
deleted if its ID number does not have a match on a format 9 car! or if its ID 
number duplicates that of another format 10 card. 

Columns 5-10 (6 column numeric field): Square, feet per facility unit. 

Columns 11-16 (6 column numeric field): Maintenance manhours per square foot 
per month. 

Format 11: Training Administrative, Base Operating Support^ and Medical 
Manpower Inputs Card Limit = 3 - This card contains inputs required for the deri- 
vatiori of training administrative, base operating supports and medical manpower. 

Training administrative personnel perform training-related overhead functions 
not otherwise accounted for. This includes functions at/the school-level 

♦ 

(Administrative Affairs Office), at the department level; (Administrative Section, 
Requirements Unit, and Instruction and Measurement Unit), and at the branch level 
(branch administration). It does not include the functions provided by the Training 
Services Division or the Instructor Training Division (school level) or the 
Curriculum Unit (department level), however. 

Base operating support personnel perform the following base housekeeping and 
service functions: supply, transportation, security police, material staff, comp- 
troller, personnel staff, food, commissary, housing, laundry and dry cleaning, 
recreation, education, transient aircraft maintenance, and general (not course 
specific) facility maintenance. Medical personnel provide medical and dental care 
tq military personnel. ^ 

Columns 3-4 (2 column numeric field): Personnel Type: Numeric personnel type 
code as follows: 

, 13 = Training Administrative 
^/ , M,- 14 = Base Operating Support 

\ ' , "15 = Medical 
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Columns 5-6 (2 column numeric field): Set to 01. 

Columns 7-11 (5 column numeric field): Number of fixed manyears per course for 
support services. 

Col umns .12-16 (5 column numeric field): Number of variable manyears per Type A 
manyear for support services.. . ,^ 

Columns 17-21 (5 column numeric field): Number of variable manyears per Type B 
manyear for support services. 

Typical values for these parameters are shown below. Type A manyears are the 
sum of student, instructor, and curriculum personnel manyears. Type B manyears 
are the military TOY student manyears. 

FIXED MANYEARS VARIABLE MANYEARS VARIABLE flANYEARS 
_ PER CO URSE PER TYPE A flU'YEAR PER TYPE^B MANYEAR 

Training Administrative 3 .05 

Base Operating Support 0 .08 .035 

Medical . • 0 .02 .005 

Format 12: Computer Management System Input s Card Limit = 3 - The three hard- 
ware items of the computer management system are: 

CLASS ITEM 

01 ^ Mainframe (CPU and Main Memory) 

02 Management Terminals 
03' , Auxiliary Storage 

For each of these, the following data are entered: 
0 Hardware class: as designated above 
0'. Hardware name: user selected name 

0 Capacity of un-it: for mainframe and management terminals, this is the 
number of CMI transactions per hour; f6r the auxiliary storage it is the 
number of student data sets (complete records of one student) that can be 
stored. 

0 Yearly cost of unit: To^tal yearly cost of the unit; if leased or purchased, 
rraintenance costs should be included. 

Column:; 3-5 (one column field): Hardware class. 

Columns 6-35 (30 column alphanumeric field):- Name of hardware device. 
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Columns 36-40 (5 column numeric field): Capacity of unit. 
Columns 41-46 (5 column numeric field): Yearly cost of unit. 

Format 13: Optional Officer/Airman/Civilian Distribution Overrides - This card 
provides the planner the option of overriding stored off icer/airman/ci vil ia.; distri- 
buti'on percentages. It is one of^two input formats on which blanks and zeroes ar^e 
differentiated. Thus, the planner needs to make entries only in the fields of those 
personnel types whose distribution he wishes to override; the other fields should be 
left blank. If the override values for a given personnel type do not total 100, 
then the model reinstates the stored values. The stored values are as follows: 





Officer 

% 


Ai rman 

% 


Ci vil ian 

% 


Curriculum 


0 


63 


37 


Hardware Maintenance' > 


2 


72 


26 


Facility Maintenance 


,0 


100 


0 


Training Administrative 


6 


39 . 


55 


Base Operating Support 


2 


64 


34 


Medical 


-20 


80 » 


0 



Columns 3-11 (three 3 column numeric fields): 
*i/rman/civilian distribution. 

Columns 12-20 (three 3-column numeric fields); 
officer/airman/ci vil ian distribution. 

Columns 21-29 (three 3-column numeric fields): 
officer/airman/ci vil ian distribution. 

Columns 30-38 (three 3-col uma numeric fields): 
power officer/airman/civilian distribution. 

Columns 39-47 (three 3-column numeric fields): 
officer/airman/ci vil ian distributioji. 

Columns 48-56 (three 3-column numeric fields): 
civilian distribution. 



Curriculum manpower officer/ 
Hardware maintenance manpower 
Facility maintenance- manpower 
Training administrative man- 
Base operating support manpower 
Medical manpower officer/airman/ 
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Format 14: Optional Miscellaneous Overrides - This card allows the planner to 
override miscellaneous time and cost factors which have been stored as part of the 
program. It is the second of two input formats on which blanks and zeroes are 
^differentiated. Thus, as before, the planner needs to 'make entries only in the 
fields of those factors which he wishes to override; the other fields should be left 
blank. The stored values are as follows: 

Available Productive Manhours/Monch 

Hardware Maintenance Personnel 121 

All other Personnel Types 142 

Avecage Number of Training Days per Month 21.7 

Classroom Training Hours per Student per Day 7 

Miscellaneous Supply Cost per Manyear ($) -11'2 
P,CS Cost per Move ($) 

Officer 879 

i ' Airman 224 

C^ivilian 879 
TDY Expense {$) 

One-Way Transportation 85 

Destination per Diem 

Officer 11 * . 

Airman 4 

Civilian 21 

Columns 3-10 (two 3-column numeric fields): Available productive manhours per 
month. Most AF training personnel have 142 hours per month available to pe,- 
fbrm productive work (174 assigned hours less 32 hours for leave, medical, 
training,, and organizational duties). Hardware maintenance personnel, however, 
must spend part of that 142 hours waiting for parts and tools. This wailing 
time is estimated at 15% of available hours, thus reducing hardware maintenance 
available productive manhours per month to 121. 

Columns 11-14 (4 column numeric field): Average number of training days per 
month. 

Columns 15-18 (4 column numeric field): Classroom training hours per student 
per day. This' value should coincide with the value listed on Worksheet 1. 
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Columns 19-23 (5 column numeric field): Miscellaneous supply cost per student 
per manyear: The cost of personnel supplies ^(food, clothing, etc.) and general 
office supplies (paper, pencils, forms, etc.) per manyear (students and 
permanent party). 

Columns 24-35 (three 4~column numeric fields): PCS v .rmanent Change of 
Station) cost per move ($): Includes transportation of personnel and depen- 
dents; shipment and/or storage of household goods; and mileage per diem and 
subsistence allowance while in travel status. 

Columns 36-39 (4 column numeric field): Average TDY one-way transportation 
cost ($): Average cost of transportation and 'travel per diem to or f^^om 
technical training center. 

Columns 40-48 (three 3-column numeric fields): TDY destination per diem ($): 
Average student cash entitlement while at technical training centers, after 
deductions for quarters and messing (when available). 

Format 15: Optional Pay and Allowance Overrides - This card permits the planner 
to override the stored pay and allowance factors. The stored values are listed in 
Figure 10. ^ . - 

Columns 3-4 (2 column numeric fields): Personnel designator: Numeric personnel 
designator code as follows: 

1 = Officer 

2 = Airman ' 

3 = Civilian - GS 

4 = Civilian - WB 

5 = Non-DoD 

Use of a code outside the range 1 through 5 will result in the deletion of that 
card. 

Columns 5-6 (2 column numeric field): Paygrade: Standard U.S. Government 
numeric paygrade codes. Card deletion will occur if the paygrade entered is 
not within the range of permissible paygrades for that personnel designator or 
if the personnel designator/paygrade combination duplicates that of another 
card. 
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Personnel Designator Permissible Paygrade Range 
Officer 0 or blank, 1-10 

Airman 0 or blank, 1-9 

Civilian - GS 0 or blank, 1-18 

Civilian - WB 0 or blank, 1-14 

Non-DoD 0 or blank, 1-20 

Columns 7-11 (5 column numeric field): Annual pay and allowance factor ($) : 
Annual pay and allowances per manyear. 

Format 16: Lesson Descriptor Cards Card Limit = 75 - This card is used to 
describe'each lesson in the new self-paced course. Thirteen descriptors are needed 
for each lesson. . These variables are described in paragraph 4.2.2 of the report. 

Columns 3-5 (3 column numeric field): Lesson ID number; a sequencing number 
assigned by the planner. This is used only for convenience in arranging the 
lessons. It is not used in the computations. 

Columns 6-25 (20 column alphanumeric field): Lesson name; user selected name 
for lesson. 

Columns 26-30 (5 column numeric field): The number of lecture/discussion 
minutes in the conventional course (CLDM). Taken directly from the POI. 

Columns 31-35 (5 column numeric field): The number of demonstration/perform- 
ance-cognitive minutes in the' conventional course .lesson (CDPMC). Taken directly 
from the POI. 

Columns 36-40 (5 column numeric field): The nurrter of minutes of memory 
content in the lesson (MEM). Provided by the course developer. 

Columns 41-45 (5 column numeric field): The numerical rating from 0 to 5 of 
the simplicity or complexity of the memory content in the lesson (MSC). 
Provided by the course developer. 

Columns 46-50 (5 column numeric field): The number of self-check items in the 
self-paced course lesson (SCKC). A design variable. 

Columns 51-55 (5 column numeric field): The number of minutes of psychomotor 
task content in the lesson (PM). Provided by the course developer. 
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Columns 56-60 (5 column numeric field): The number of programmed instruction/ 
audiovisual minutes in the conventional course lesson (CPIAVM). Taken directly 
from the POL 

Columns 61-65 (5 column numeric field): The lumber of demonstration/performance 
psychomotor minutes in the conventional cours? lesson (CDPMP). Taken directly 
from the POL 

Columns 66-70 (5 column numeric field) r The numerical rating from 0 to 5 of 
the simplicity or complexity of the psychomotor content in the lesson (PMSC). 
Provided by the course developer. 

Columns 71-75 (5 column numeric field): The number of minutes of cognitive 
content in the lesson (COG), Provided by the course developer. 

Columns 76-80 (5 column numeric field): The number of outside assignment 
minutes in the conventional course (COUTM). Taken directly from the POL 

Format 99: Termination Card - No entries other than the "99"^in columns 1 and 
2 need be made. 

Computer Program Output Description 

The output from the computer program is illustrated in this section through the 
use of an example run from the program. Since much of the output format remains as 
it was designed. by Rand for the MODCOM Program, we have summarized their desqriptions 
as in MODIA: The Cost Model (Hess & Kantar, 1976). The user of the model will note 
that some small rounding errors occur in the totals. 

Output 1 (Figure A-l l - This is an exact echo of the input data. Card columns 
are shown on the top of the page so any misalignment of data or input is apparent. 

Output la (no figure) - This section contains error messages relating to data 
that were found to be outside the specified range, or from duplicate cards. Fatal 
errors, such as no lesson descriptors or the omission of the end of data card 
(Format 99) will also appear here. 

Output 2 (Figure ^-2 - This output lists the values assigned to each variable 
in the lesson descriptor set. Only those lessons which have passed the tests for 
acceptable range appear here. 
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Output 3 (Figure A-3) - This table lists the computed time to complete each 
Jesson for students wi^'i various student variable values. Times are shown in hours. 
Tptal course time for each time student is shown, as well as the weighted average 
time to compel te the course, and the percentage timesaving (if TPOI is given). 

Output 4 (Figure A-4) - This output lists the personnel distributions and cost 
factors^ stored as part of the model. These, may be overridden by inputting data 
(Formats' 14-1^6 J. ■ 

Output 5,: Graduat Summary (Figure A-5) - Output 5 is a time-phased summary of 
Active DutyVorce, Guard and Reserve, other DoD and Non-Dod graduates. The numbers 
reflect only tHpse students who actually graduate in that year. For example, if the 
course duration v(ere greater than one year, then no graduates would appear in year 1. 
The "in progress" column indicates the number of students who entered in years 4 and 
5 but who will not g'y;aduate until years 6 or 7. All Air Force students except those 
assigned to the Air National Guard and the Air Force Reserve are termed Active Duty 
Force. Guard and Reserve refers only to Air Force Guard and Reserve students. 
Students assigned to DoD agencies other than the AF, such as the Arrny and Navy 
(including their Guard and Reserve components) are classified as other DoD. Finally, 
students from non-DoD agencies, j'ncluding foreign governments (e.g.. Military 
Assistance Program), are termed Non-DoD. 

Output 6, Manpower Summary (Figure A-6) - Output 6 is a two part, time-phased 
summary of student and base permanent party manyears. Part 1 is a functional break- 
down covering all course personnel while Part 2 is a PCS/TDY breakdown limited to 
AF personnel . 

r 

Part 1 

Total Student Load : Total student manyears. ^ 

Instructors include all academic, reme^dial and special requirements instructors 
as well as course monitors and supervisors.. 

Curriculum Personnel determine training requirements and develop course written 
materials. Normally attached to the Curricula of a training department, these 
personnel share the task of courseware development with the course's instructors. 

Hardware Maintenance Personnel are responsible for the preventive and correc- 

f 

tive maintenance of course hardware. Media hardware maintenance is generally the 
responsibility of the Training Services Division (school level), while all other 
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types of hardware are usually serviced by the base Maintenance and Supply Group or a 
specialized maintenance squadron. 

Facilities Maintenance Personnel are responsible for the maintenance and repair 
(including custodial service) of course-related facilities. * ^ 

Training Administrative Personnel perform training-related overhead functions 
not otherwise accounted for. This includes functions at the school level (Admini- 
strative Division, Training Evaluation Division, Operations Division, and Foreign 
Military Affairs Office), at the department level (Administrative Section, Require- 
ments Unit, and Instruction and Measurement Unit), and at the branch level (branch 
administration). It does .not include the functions provided by the Training Service 
Division or the Instructor Training Division (school level) or the Curricula Unit 
(department level), however. ^ 

Base Operating Support (BOS) personnel perform the following base housekeeping 
and service functions: supply, transportation, security police, material staff, 
comptroller, personnel staff, food, commissary, housing, laundry and dry cleaning, 
recreation, education, transient aircraft maintenance, and general (not course 
specific) facility maintenance. 

Medical Personnel provide medical and dental care tO" military personnel. 

Total Base Permanent Party represents those persohnel assigned to a base for 
the purpose of performing duty in the furtherance of the mission of that base. Thus, 
students are excluded from this total. 

Total Course Manyears represents total student load plus total base permanent 
piirty. 

Part 2 . 

Active Duty Force PCS Student InaH: The number of PCS* manyears accrued by 
Active Duty Force students. Pipeline students are always assumed to be PCS while 
lateral and upgrade students are PCS only if the course duration is 20 weeks or more. 

Base Permanent Party - AF Only: Total base permanent party less non-AF instruc- 
tors. Specific program elements charged are: 
Special ized Training 
Air Force Instructors 
Curriculum Personnel 
Hardware Maintenance Personnel 
Training Administrative Personnel 



♦Permanent Change of Station means the reassignment of a military person from one 
permanent duty station to another. a-26 X2'? 
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Hospitals 

^ Medical ^Personnel assigned to hospitals 
Real Property Maintenance Activities 

Facilities Maintenance Personnel ^(course-related) 
Facility- related portion of BOS Personnel (general base housekeeping) 
Base Operations 

BOS Personnel (except facility-related portion) 
Medical Personnel assigned to dispensaries 
Active Duty Force TDY Student Load : Namber of TDY* manyears accrued by Active 
• Duty Force students including Program 8 TDY students. Pipeline students are never 
TDY while lateral and upgrade students are TDY only 'if the course duration is less 
than '20 weeks. They ^nay be assigned to any program element in Air Force Programs 
1-4 and 6^10. , 

Guard and Reserve Student Load : Number of manyears (PCS and TDY) accrued by 
Air Force Guard and Reserve students. All Guard and Reserve manyears are charged 
to Program 5. 

Total Air Force Manyears : The sum of total Program 8 manyears, Active Duty 
Force TDY student manyears, and Ruard/Reserve student manyears. 

Ou tput 7, Courseware, Hardware, and Facility Requirements (Figure A-7) - Output 7 
is a summary of key courseware, hardware, and facility inputs and computet require- 
ments. 

Output 8, Functional Cost Summary (Figure A-8j - Output 8 is a breakdown of total 
course costs by function. 

Investment Costs 

, Courseware Procurement includes four separate costs: (1) the initial cost of 
producing the master from which all subsequent copies will be made','(2) the cost of 
reproducing and packaging the required number of copies; (3) the annual cost of 
revisipn due to change in course content; and (4) the annual cost of replacement due 
to loss, damage, and normal wearout. It does not include the cost of courseware 



**X6mporary Dut^ means duty at a location other than the permanent duty station at 
which a member performs temporarily under orders which provide either reassignment 
to the old permanent station or assig^iment to a new permanent station. 
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development such as script writing and editing, which is accounted for within 
curriculum manpower pay and allowances. 

Hardware Procurement embraces the following cost elements: (1). the cost of 
purchasing the incremental number of units required each year, including initial 
replacement part stocks; (2) credits for any surplus items returned'to the Stock 
Fund; ^and (3) the annual cost of replacing those units worn out during the year. 
The annual cost of replenishment repair parts is accounted for elsewhere. Negative 
nunibers indicate a dominance of Stock Fund credits. 

Facility Construction represents the total cost of constructing and/or modify- 
ing facilities required for course use. 



Operating Costs 

Pay and "Allowances provides for all officer, airman, and civilian pay and 
allowances. The elements accounted for are as follows; 



Officers 

Basic Pay 

Special Pay 

Basic Allowance for 
Quarters 

Basic Allowance for 
Subsistence 

Uniform Allowance 

Family Separation 
Allowance 

Separation Payments 

Social Security Tax - 
Employer's Contribution 



Airmen 

Basic Pay 

Special Pay 

Proficiency Pay 

Reenlistment Bonus 

Basic Allowance for Quarters 

Clothing Allowance 

Separation Payments 

Social Security. Tax - Employer's 
Contribution 



Ci vi 1 i an 

Basic Pay 
Life Insurance 
Health Benefits 
Terminal Leave 
Workman's Compensation 
Civilian Retirement 
Overtime 



PCS Costs include the expenses incident to the permanent change of station of 
students and instructors: transportation of personnel and dependents; shipment and/ 
or storage of household goods; and mileage per diem and subsistence allowances while 
in travel status. 

TDY Costs include commercial transportation, car rental, mileage allowances and 
tools, per diem, and incidental expenses in. ,rred by students in authorized travel 
status. 
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I nstructor Training is composed of instructor educatiori training costs (the 
cost of the six-week instructor prerequisite course on education principles) and 
initial factory training costs (the cost of providing the initial instructor cadre 
with necessary specialized equipment or system training). 

Miscellaneous Operating Costs consists of three distinct elements: 

1. Computer Service Charges: Any computer expenses incurred as a result 
of using computer based instruction. 

2. Hardware Replenishment Repair Parts: The cost of purchasing 
miscellaneous hardware repair parts. 

3. Miscellaneous Supplies: The cost of personnel supplies (food, clothing, 
etc.) and general office supplies (paper, pencils, forms, etc.). 

The planner should carefully inspect each type of output to assure himself 
that the answers are reasonable. Areas where large costs appear may focus further 
planning effort to reducing these costs. 
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